Category

网志

Safety, Performance and Innovation: Rust in Hyperledger Sawtooth

By | Blog, Hyperledger Sawtooth

Hello, fellow Rustaceans and those curious about Rust. The Hyperledger Sawtooth team is using Rust for new development, so these are exciting times for both Rust and Hyperledger Sawtooth. Rust is a new language that is quickly growing in popularity. The Hyperledger Sawtooth community is using Rust to build components to give application developers and administrators more control, more flexibility, and greater security for their blockchain networks. This blog post will give an overview of some of the new components being built in Rust.

Hyperledger Sawtooth was originally written in Python, which was a good choice for initial research and design. In 2018, the Sawtooth team chose the Rust language for new development. A key benefit is that Rust supports concurrency while also emphasizing memory safety. Several new core components, transaction processors, and consensus engines have already been written in Rust.

Compared to Python, Rust’s most noticeable feature is the expressive type system, along with its compile-time checks. Rust has ownership and borrowing rules to guarantee at compile time that an object has either only one mutable reference of an object or an unlimited number of immutable references. This feature of Rust forces the developer to account for all possible error and edgecases, making our interfaces more robust as we design them.

The validator’s block validation and publishing components are a good example of our recent interface changes. Before release 1.1, these components were heavily tied to PoET, the original consensus in Hyperledger Sawtooth. In addition, they were largely synchronous, where committing a block started the process of building a new block to publish. As we implemented the consensus engine interface, we took the opportunity to rewrite these components in Rust, which helped us to separate them more cleanly. Now there are three separate asynchronous tasks—block validation, block commit, and block publishing—that share a small amount of information. For example, the block publishing component is informed when batches are committed so that it can take them out of a pending queue, but none of the tasks starts either of the other tasks. For more information, see the block validation and block publishing components in the sawtooth-core repository.

This clean separation of tasks allows the new consensus interface to function correctly and makes it easier to develop new consensus engines. The Sawtooth team has already written two new engines in Rust:  Sawtooth PBFT and Sawtooth Raft (which uses the PingCap raft library, raft-rs). The Sawtooth team is proud of the work we have done on these consensus engines, and the flexibility it provides Sawtooth community members who are building a blockchain application.

Rust also excels in its support for compiling to WASM, which can be used as a smart contract. Hyperledger Sawtooth already had Seth, which supports running Ethereum Solidity smart contracts using a transaction processor, but now has Sawtooth Sabre, a transaction processor that runs a WASM smart contract that is compiled from Rust to the WASM target. Sawtooth Sabre includes an innovative feature: using registries for namespaces and contracts. The namespace registry lets administrators control what information a contract can access. The contract registry lists versions of the contract, along with a SHA-512 hash of the contract, giving application developers confidence that the correct contract is registered. Sabre supports API compatibility with the Sawtooth Rust SDK, so developers can write a smart contract that can run either within Sabre or natively as a transaction processor, depending on the deployment methodology.

Rust has also influenced how changes to Hyperledger Sawtooth are handled. Our new RFC process is modeled after Rust’s RFC process, which provides a community-oriented forum for proposing and designing large changes. The Hyperledger Sawtooth team has put effort into a community-oriented design process at sawtooth-rfcs. The consensus API RFC is a good example: The guide-level explanation clearly lays out the purpose and reasoning behind the new component, then has a reference-level explanation of the technical details needed to guide implementation. The Sawtooth RFC process has been a good way to involve the larger Sawtooth community in driving the design and implementation of Sawtooth.

What’s next for Rust in Sawtooth? In 2019, the Sawtooth team is rewriting the remaining Sawtooth validator components in Rust. That means the networking and transaction processing components will be getting an overhaul. Expect that the networking components will be redesigned. The transaction processing components will have minor changes internally, while keeping a stable API. In both cases, there will be an increase in performance and stability thanks to Rust.

Come join the Hyperledger Sawtooth community in 2019 by writing your own transaction processor in Rust or even a consensus engine. Get in touch on the #sawtooth channel on RocketChat.

To learn more about Rust in Hyperledger Sawtooth, check out our recent changes:

 

About the Author:

Boyd Johnson is a Software Engineer at Bitwise IO who has worked on many core components of Hyperledger Sawtooth, including transaction processing components in Python and block validation and block publishing components in Rust. While originally a committed Pythonista, he has oxidized into a Rustacean.

Developer showcase series: John Greenhow, Peloton Blockchain

By | 网志, Developer Showcase, Hyperledger Composer, Hyperledger Fabric

Back to our Developer Showcase Series to learn what developers in the real world are doing with Hyperledger technologies. Next up is John Greenhow of Peloton Blockchain.

What advice would you offer other technologists or developers interested in getting started working on blockchain? 

To anyone that wants to get started working on blockchain, my advice is to just start. As we set out, our team had to go through a headspace change similar to moving from assembly to object-oriented.

We picked a platform, picked a simple goal and ran through the installation and a test. More than once if necessary (I installed and setup Fabric twice). As soon as you physically can, look at the files on disk or in the database to see how they change – it will demystify things much more quickly.

We started off with Hyperledger Composer. This was a fantastic way in for us. It allowed us to model the business network we had envisioned and to write and test the necessary logic very quickly, all while going back to check bytes-on-disk to see how things worked under the hood.

My other piece of advice starts with an old saying, “when you’re holding a hammer, everything looks like a nail”. There will be use cases out there to which blockchain is not well suited as a solution, and that’s OK.  Spend time looking thinking about eco-systems and business networks and where the cost of a decentralized solution is really going to make a difference to everyone in the network.  It will pay off in the long run.

Give a bit of background on what you’re working on, and let us know what was it that made you want to get into blockchain?

Peloton Blockchain has developed software and a network for the tokenisation, issuance, discovery, listing, secondary trading and redemption of financial instruments and non-financial assets – in a way that is compliant with regulatory jurisdictions around the world.   

We can offer efficiencies in primary markets, new distribution channels for issuers and a whole new class of products for exchanges. We have 1 patent pending with 2 further patents in the pipe. Exciting times!

We’ve developed 3 products:

  • Issuer/Arranger System – this SaaS supports the tokenisation of products and publishing of new tokens or securities to a blockchain network; no development or scripting is required for new securities.
  • Blockchain Network – based on Hyperledger Fabric, this is a private, secure and scalable network designed to provide unparalleled breadth of distribution to issuers and lead arrangers; chain-code manages compliance and the full security lifecycle – we’ve built RESTful APIs to facilitate connectivity into crypto-exchanges and secure “nodes” for deployment.
  • Exchange System – an exchange trading platform designed for primary and secondary market trading of tokens and securities; we don’t (and won’t!) run a market ourselves, but we do provide the software and expertise if it’s needed to setup secondary markets.

Blockchain in general, and specifically Hyperledger Fabric, provided the best platform upon which we could build. The ability to deploy software to many different organizations and yet retain a trusted view of a ledger of security transactions and ownership opens a utopian vision for many in the capital markets. Blockchain is simply the perfect tool for this job.

What project in Hyperledger are you working on? Any new developments to share? Can you sum up your experience with Hyperledger?

We initially started on Composer and then moved to Fabric as we moved towards a production deployment model. We were sorry to leave Composer behind as it has so many great features – the modeling language and API generation chief among them.

Moving to Fabric involved a re-write into Golang, and we were really surprised by just how much code was needed to port what had been a fairly small project in Composer. I don’t say this to denigrate Fabric but to accentuate the power of Composer.

Our focus now is building out a production network. As well as deploying nodes to participating market operators, issuers and arrangers, we are hosting several nodes around the world to form a reliable back-bone to the network. To do this, we’ve utilized the AWS Elastic Kubernetes Service (EKS) and this has been a powerful tool in ensuring scalability.

We’re also looking ahead to production operations, and building web services and management consoles to automate and coordinate everything from upgrades to onboarding new network participants.

What do you think is most important for Hyperledger to focus on in the next year?

There are several routes available by which Hyperledger can continue to lower barriers to entry for new teams and projects. These include:

  1. A move to commoditization – installation, configuration and operation of databases, message buses, caches and many other complex software packages is out-of-the-box today. Experience in the field will contribute to incremental releases that continually shorten the time taken to be productive.
  2. Best of breed flagship – rolling together the (stable) killer features of different Hyperledger projects into a commodity enterprise project could reduce the risk stemming from dependency upon a single project and provide a single point of entry into the Hyperledger world for teams focused on solutions rather than platform evaluation. An example might include Fabric support for Composer models (or perhaps better, Composer cross-compilation to Fabric-ready Golang) packaged with Cello features pre-configured for particular use-cases.

Ongoing community out-reach, profile pieces and education remain invaluable and the more the better!

What’s the one issue or problem you hope blockchain can solve?

What really fascinates me about blockchain is that it rarely seems to be the most efficient or low-cost solution available. I’ve had endless conversations with technologists who see a simple, centralized solution as faster…easier…cheaper!

From the perspective of a single organization, this is true. Where the technology comes into its own is in making whole eco-systems of organizations more effective. Blockchain isn’t growing the one slice, it’s growing the whole damn pie.

How successful blockchain projects (in terms of adoption) are seems to depend on the benefits brought to all participants, rather than to a central orchestrator. To the central orchestrator, data is the new oil. Is that still the case if everyone has a copy of that data?

Seen through this filter, there is an implication of more broadly available efficiencies to many industries. This can only be good for economic growth in both developed and high potential markets.

Our project at Peloton Blockchain focuses directly on this, building highly connected networks of issuers, arrangers and exchanges to the benefit of all.

Where do you hope to see Hyperledger and/or blockchain in 5 years?

Ultimately blockchain is an application infrastructure layer, much like an RDBMS or messaging bus.

While there is (quite rightly) a lot of research and energy going into blockchain today, the hope is that the layer will become commoditized and that the focus will shift to the solutions that this enables for business networks.

Hyperledger already has projects that are well on the way to making this a reality and we’re really excited to see this continue.

What is the best piece of developer advice you’ve ever received?

Test your code!

Floating the Sawtooth Raft: Implementing a Consensus Algorithm in Rust

By | 网志, Hyperledger Sawtooth

The 1.1 release of Hyperledger Sawtooth includes official support for a new consensus API and SDKs. These tools, covered in an earlier blog post, open up new possibilities for Sawtooth developers, giving them the power to choose a consensus algorithm that best suits their needs. With support for Proof of Elapsed Time (PoET) and Dev mode consensus engines already available, we decided to expand the platform’s repertoire to include a wider variety of engines and support a broader array of features and use cases. The first of these new engines implements the Raft consensus algorithm. This blog post gives a brief overview of the Raft algorithm, explains our decision to implement it, and takes a quick look at the development of the Raft consensus engine.

What Is Raft?

Originally developed by Diego Ongaro and John Ousterhout at Stanford University in 2013, Raft is designed to be an easy-to-understand, crash fault tolerant consensus algorithm for managing a replicated log. Its primary goal is understandability, since most deterministic consensus algorithms previously developed were convoluted and difficult to grasp. Raft provides crash fault tolerance, allowing a network to continue to make progress as long as at least half of the nodes are available.

Raft has the following key characteristics that set it apart from many other consensus algorithms:

  • Strong leadership: Networks elect a leader that is responsible for making progress
  • Non-forking: Unlike lottery-based algorithms, Raft does not produce forks
  • Closed membership: Raft does not support open-enrollment, but nodes can be added and removed by an administrator
  • Fully peered: All nodes must be peered with all other nodes
  • Crash fault tolerant: Raft does not provide Byzantine fault tolerance, only crash fault tolerance

Raft’s leader-follower model is a direct result of the emphasis placed on simplicity and understandability. With a single node controlling the progress of the log, no forks arise so no extra logic is needed to choose between forks. The leadership model has important implications for other aspects of the algorithm. Because a majority of nodes must agree on the elected leader and on all network progress, membership must be semi-fixed to prevent disjoint majorities. This means that Raft networks do not support open enrollment; membership in the network is restricted and can only be modified by a privileged user.

Raft consensus networks must also be fully peered—with each node connected to all other nodes—because messages need to be passed between all nodes. Furthermore, because a large volume of messages is required for the algorithm to work, larger Raft networks perform slower than smaller networks. If high performance is important, Raft would be best used for smaller networks—usually 10 nodes or fewer.

Lastly, Raft is limited to just guaranteeing crash fault tolerance, not Byzantine fault tolerance. This makes the Raft algorithm ill-suited for networks that are subject to Byzantine faults such as groups of malicious nodes. For more information about the Raft algorithm, please see the original Raft paper and the Raft website.

Why Raft?

Raft was our choice for the first algorithm with the new consensus API for several reasons. First, it is very different from PoET. Where PoET is a forking, lottery-style algorithm, Raft is leader-based and non-forking. This allowed us to not only demonstrate the flexibility of the Sawtooth consensus API, but also to make an algorithm available that is well-suited for situations that an algorithm like PoET is not a good fit for.

Also, Raft is an inherently simple and easy-to-understand algorithm. This made it trivial to adapt to Sawtooth and also made it an excellent example for developing other engines.  Furthermore, we took advantage of an existing high quality implementation of Raft in the Rust programming language called raft-rs.

However, Raft lacks Byzantine fault tolerance. Therefore, we are also working on a PBFT consensus engine that is suitable for consortium-style networks with adversarial trust characteristics.

The Implementation

The raft-rs library, developed by PingCAP, provides almost everything we needed to implement a consensus engine based on the Raft algorithm; it provided a class representing a Raft “node” with a handful of straightforward functions for “driving” the algorithm. The folks at PingCAP wrote an excellent blog post explaining how they implemented this library, so we will not duplicate their efforts here.

Our only major extension to the raft-rs library is a stable storage mechanism, since the library only provided in-memory storage. This extension is required to ensure that Sawtooth nodes can restart in the event of a crash or arbitrary shutdown. If you would like to see the end results, all of the code that follows can be found in the Sawtooth Raft GitHub repository and the Rust SDK.

Defining the Engine

The first step in creating a consensus engine with the Rust SDK is to implement the Engine trait:

pub trait Engine {

    /// Called after the engine is initialized, when a connection

    /// to the validator has been established. Notifications from

    /// the validator are sent along `updates`. `service` is used

    /// to send requests to the validator.

    fn start(

        &mut self,

        updates: Receiver<Update>,

        service: Box<Service>,

        startup_state: StartupState,

    ) -> Result<(), Error>;

 

    /// Get the version of this engine

    fn version(&self) -> String;

 

    /// Get the name of the engine, typically the algorithm being

    /// implemented

    fn name(&self) -> String;

}

Raft’s Engine implementation is in engine.rs. The start method is the main entry point. In Raft—as well as most consensus engines—three main tasks need to be performed here: loading configuration, creating the struct(s) that contain the core logic, and entering a main loop.

Loading Configuration

For Raft, loading configuration consists primarily of reading a few settings that are stored on-chain. We do this by making a call to the load_raft_config function in config.rs:

// Create the configuration for the Raft node.

let cfg = config::load_raft_config(

    &local_peer_info.peer_id,

    chain_head.block_id,

    &mut service

);

info!("Raft Engine Config Loaded: {:?}", cfg);

let RaftEngineConfig {

    peers,

    period,

    raft: raft_config,

    storage: raft_storage

} = cfg;

The settings are loaded by calling the get_settings method in the consensus service, with the chain head provided in the startup_state:

let settings_keys = vec![

    "sawtooth.consensus.raft.peers",

    "sawtooth.consensus.raft.heartbeat_tick",

    "sawtooth.consensus.raft.election_tick",

    "sawtooth.consensus.raft.period",

];

 

let settings: HashMap<String, String> = service

    .get_settings(block_id, 

        settings_keys.into_iter().map(String::from).collect())

    .expect("Failed to get settings keys");

 

Some of these settings are optional, so defaults are used if they’re unset.

Creating the Raft Node

Once the configuration is loaded, we create the Raft node that contains the main logic of the algorithm:

// Create the Raft node.

let raft_peers: Vec<RaftPeer> = raft_config.peers

    .iter()

    .map(|id| RaftPeer { id: *id, context: None })

    .collect();

let raw_node = RawNode::new(

    &raft_config,

    raft_storage,

    Raft_peers

).expect("Failed to create new RawNode");

 

let mut node = SawtoothRaftNode::new(

    local_peer_info.peer_id,

    raw_node,

    service,

    peers,

    Period

);

The RawNode struct is provided by the raft-rs library; it contains the logic for the Raft algorithm itself and provides methods for SawtoothRaftNode to direct it. The SawtoothRaftNode, found in node.rs, defines six methods that are called by the consensus engine:

  • on_block_new is called when the validator notifies the engine that it has received a new block
  • on_block_valid is called when the validator notifies the engine that it has validated a block
  • on_block_commit is called when the validator notifies the engine that it has committed a block
  • on_peer_message is called when one node’s consensus engine sends a message to another
  • tick is used to move the Raft algorithm forward by one “tick”
  • process_ready contains much of the logic that changes the state of Raft

The first four methods (on_block_new, on_block_valid, on_block_commit, and on_peer_message) will be defined for the majority of consensus engines since they handle important messages that are delivered by the validator. The last two methods (tick and process_ready) are specific to Raft; other consensus engines will likely have different methods to handle the logic of the engine.

Entering the Main Loop

With a Raft node created and ready to handle updates, we enter the main loop of our consensus engine:

let mut raft_ticker = ticker::Ticker::new(RAFT_TIMEOUT);

let mut timeout = RAFT_TIMEOUT;

 

// Loop forever to drive the Raft.

loop {

    match updates.recv_timeout(timeout) {

        Err(RecvTimeoutError::Timeout) => (),

        Err(RecvTimeoutError::Disconnected) => break,

        Ok(update) => {

            debug!("Update: {:?}", update);

            if !handle_update(&mut node, update) {

                break;

            }

        }

    }

 

    timeout = raft_ticker.tick(|| {

        node.tick();

    });

 

    if let ReadyStatus::Shutdown = node.process_ready() {

        break;

    }

}

Raft’s main loop performs three main tasks. First, check if there are any updates that have been sent to the engine by the validator. If there is an update, handle it by calling the appropriate method of the SawtoothRaftNode:

fn handle_update<S: StorageExt>(node: &mut SawtoothRaftNode<S>, 

  update: Update) -> bool

{

    match update {

        Update::BlockNew(block) => node.on_block_new(block),

        Update::BlockValid(block_id) =>

            node.on_block_valid(block_id),

        Update::BlockCommit(block_id) => 

 node.on_block_commit(&block_id),

        Update::PeerMessage(message, _id) => 

 node.on_peer_message(&message),

        Update::Shutdown => {

            warn!("Shutting down");

            return false

        },

 

        update => warn!("Unhandled update: {:?}", update),

    }

    true

}

Second, move the Raft algorithm forward by one “tick” at a regular interval, using the Ticker object defined in ticker.rs and a call to the node’s tick method. This “tick” roughly corresponds to progress in the Raft algorithm itself.

Finally, call the node’s process_ready method, which checks the state of the Raft algorithm to determine if it needs to take any actions as a result of the last “tick”.

Starting the Engine

Once the consensus engine itself has been defined, starting it up and connecting it to the validator is easy. In the main function of main.rs, all we need to do is simply determine the validator’s endpoint (using a command-line argument in Raft), instantiate the engine, and start it using the SDK’s ZmqDriver:

let raft_engine = engine::RaftEngine::new();

 

let (driver, _stop) = ZmqDriver::new();

 

info!("Raft Node connecting to '{}'", &args.endpoint);

driver.start(&args.endpoint, raft_engine).unwrap_or_else(|err| {

    error!("{}", err);

    process::exit(1);

});

See for Yourself!

Want to try running a Sawtooth network with Raft consensus? Check out the Raft source code on GitHub as well as the Sawtooth Raft documentation for all you need to get started.

For more on the consensus API and developing your own consensus engine for Hyperledger Sawtooth, take a look at our previous blog post.

 

About the Author

 

Logan Seeley is a Software Engineer at Bitwise IO. He has been involved in a variety of Hyperledger Sawtooth projects, including the development of the consensus API, Sawtooth Raft, and Sawtooth PBFT.

Introducing Hyperledger Fabric 1.4 LTS!

By | 网志, Hyperledger Fabric

The pace of blockchain development and deployment is accelerating. Fortunately, so too is the scale and maturity of the community and technology at the core of Hyperledger Fabric. With an eye towards the growing rollout of production Fabric networks, we are pleased to introduce Fabric v1.4 LTS, our first long term support release.

The Fabric developers have been working with network operators and application developers to deliver v1.4 with a focus on production operations and developer ease of use. Key new production-focused features of Fabric fall into four key buckets:

  • Serviceability and Operations: As more Hyperledger Fabric networks get deployed and enter a production state, serviceability and operational aspects are critical. Fabric v1.4 takes a giant leap forward with logging improvements, health checks, and operational metrics. Along with a focus on stability and fixes, Fabric v1.4 is the recommended release for production operations. Future fixes will be delivered on the v1.4.x stream, while new features are being developed in the v2.0 stream.
  • Improved programming model for developing applications: Writing decentralized applications has just gotten easier. Programming model improvements in the Node.js SDK and Node.js chaincode makes the development of decentralized applications more intuitive, allowing you to focus on your application logic. The existing npm packages are still available for use, while the new npm packages provide a layer of abstraction to improve developer productivity and ease of use. We have also provided a comprehensive business scenario and tutorial to get you started with the new developer experience.
  • Enhanced data privacy: Data and transaction confidentiality has been a key driver for Fabric development since v1.2. With this new release, we have added two new enhancements: 1) peers for organizations that are added to private data collections can now retrieve the private data for prior transactions to which they now are entitled, and 2) automatically enforce access control within chaincode based on the client organization collection membership without having to write specific chaincode logic.
  • Hand-on tutorials: Commercially focused training to help developers move up the Fabric learning curve quickly and efficiently to speed adoption and deployment.

Complete details of the new features can be found in the What’s New documentation and release notes.

Hyperledger Fabric’s First Long Term Support Release.

As noted, Hyperledger Fabric v1.4 LTS marks our first long term support release. This is a critically important development for those beginning to deploy Hyperledger Fabric solutions into production and is a reflection of the confidence that the Fabric maintainers have in this latest release.

Our policy to date has been to provide bug fix (patch) releases for our most recent major or minor release until the next major or minor release has been published. We plan to continue this policy for subsequent releases. However, for Hyperledger Fabric v1.4 LTS, the Fabric maintainers are pledging to provide bug fixes for a period of one year from the date of release (Jan 10). This will likely result in a series of patch releases (v1.4.1, v1.4.2, …), where multiple fixes are bundled into a patch release.

If you are running with Hyperledger Fabric v1.4 LTS, you can be assured that you will be able to safely upgrade to any of the subsequent patch releases. In the advent that there is need of some upgrade process to remedy a defect, we will provide that process with the patch release.

If you’d like to join the community or learn more, you can find more information here:

Chat: #Fabric in Hyperledger Chat

Docs: Fabric 1.4 Documentation

Code: Fabric Github

Issues & Roadmap: Fabric JIRA

Website: Hyperledger Fabric Homepage

Thanks for reading about our newest Fabric release. We encourage developers to try these new features out and give us feedback!

Developer showcase series: Arielle Telesmanic, Scroll

By | 网志, Developer Showcase, Hyperledger Composer, Hyperledger Fabric

Back to our Developer Showcase Series to learn what developers in the real world are doing with Hyperledger technologies. Next up is Arielle Telesmanic of Scroll.

What advice would you offer other technologists or developers interested in getting started working on blockchain? 

Blockchain is quite different from its centralized information system’s counterparts. It is very important to understand the fundamentals of Blockchain Technology and design a Blockchain ecosystem that is both cost effective and efficient to provide the favorable conditions needed to support industry migration or a public user base depending on intended use. Before getting to carried away at the drawing board learn about smart contracts, consensus models, blockchain states and the nature of the peer to peer distribution of data within a blockchain system. Ask yourself, “Can this scale? Is this a better alternative to traditional systems? What information do I need to record and how is this information being distilled to the right participants?

Give a bit of background on what you’re working on, and let us know what was it that made you want to get into blockchain?

Currently, I am working on the logistics and system design of three blockchain projects focused on ensuring data integrity in supply chains and streamlining data recovery in the advent of a data breach or leak. To complement the data recovery project, we are also developing a main net to strengthen the modularity and ease of use in migrating to a Blockchain. Our efforts include optimizing transaction speeds, side chain communication and a method to promote efficient change-management.

My interest in blockchain peaked as I noticed the advancements in quantum computing have been Signiant. Quantum computing’s ability to decrypt cryptosystems used in industry is concerning and there needs to be more development on quantum-resistant encryption and data verification/handling.

What project in Hyperledger are you working on? Any new developments to share? Can you sum up your experience with Hyperledger?

Currently, we are working with Hyperledger Fabric and Composer. Hyperledger’s framework uses chaincode which allows us to work with different states of the blockchain per transactions triggers by different applications. Out of the box, Hyperledger has promise in aiding the permissioned access to states of the ledger which is not typically supported in traditional Blockchain solutions. More information to follow on our integration efforts and contribution to the Hyperledger framework on Https://scroll.network

As Hyperledger’s incubated projects start maturing and hit 1.0s and beyond, what are the most interesting technologies, apps or use cases coming out as a result from your perspective?

Although Blockchain technology originated as a fintech solution, I am more interested in identity management and data quality assurance use cases (specifically data validity). The world is data-driven. Whether data is used to report manufacturing defects, drive patient care plans or execute a transaction that involves Personally Identifiable Information, it is of utmost importance that data is accurate.

Making Dynamic Consensus More Dynamic

By | 网志, Hyperledger Sawtooth

In October 2017, the Hyperledger Sawtooth team started to implement a new consensus algorithm for Hyperledger Sawtooth. We wanted a voting-based algorithm with finality, which is very different from the Proof of Elapsed Time (PoET) consensus algorithm that has been closely associated with Hyperledger Sawtooth since its start. This project presented a number of challenges and opportunities.

The greatest challenge in implementing this new consensus algorithm with Sawtooth was in breaking apart an architecture that has been heavily influenced by a lottery-based consensus algorithm with forking. A lot of refactoring and architectural work went into making both voting-based and lottery-based algorithms work well with Sawtooth.

However, the opportunities that we discovered from this effort made overcoming these challenges more than worth it. We designed a new consensus API that simplifies the process of adding new consensus algorithms while continuing to support the existing PoET and Dev mode consensus algorithms. We completed the first prototype validator with consensus API support in July 2018. Since then, we have been able to implement two new voting-based consensus algorithms for the Hyperledger Sawtooth platform: Raft and PBFT.

We are pleased to announce that the Sawtooth 1.1 release supports the new consensus API. This release also includes consensus SDKs to make it easier to implement new consensus algorithms.

Consensus as a Process

The new consensus architecture moves consensus functionality to a separate process, called a consensus engine, and provides an API for each consensus engine to interact with the validator.

Moving the consensus functionality to a separate process allows consensus engines to be implemented in a variety of languages. Currently, SDKs are available for Python and Rust and have been used to create the consensus engines for PoET, PBFT, and Raft.

Multi-language support is important beyond providing a choice for implementing a new consensus engine. This support makes it much easier to reuse existing implementations of consensus algorithms. For example, the Sawtooth Raft consensus engine is built on the pingcap/raft-rs library. We were able to easily integrate this well-regarded Raft library, which is itself a port from the widely-used etcd Raft library.

As SDKs for additional languages are built on top of the consensus API, it will be possible to add more and more consensus algorithms into Hyperledger Sawtooth. For example, a consensus SDK for Go would bring existing implementations such as Hyperledger Labs’ MinBFT one step closer to being compatible with Sawtooth.

Driving the Blockchain with a Consensus Engine

The consensus API is centered around a new consensus engine abstraction that handles consensus-specific functionality. A consensus engine is a separate process that interacts with the validator through the consensus API using protobuf messages and ZMQ.

The role of a consensus engine is to advance the blockchain by creating new blocks and deciding which blocks should be committed. Specifically, a consensus engine must accomplish the following tasks:

  • Determine consensus-related messages to send to peers
  • Send commands to progress the blockchain
  • React to updates from the validator

The validator continues to handle the mechanics of validation, communication, and storage for blocks, batches, and transactions. The validator must perform these tasks:

  • Validate the integrity of blocks, batches, and transactions
  • Validate the signatures for blocks, batches, transactions, and messages
  • Gossip blocks, batches, and transactions
  • Handle the mechanics of block creation and storage
  • Manage the chain head directly

New Consensus API and SDKs

The validator exposes the API for consensus engines as a set of protobuf messages sent over a network interface. This API is split into two types of interactions:

  • Service: A pair of (request, response) messages that allow a consensus engine to send commands to the validator and receive information back. For example, a consensus engine can instruct the validator to commit a block or request an on-chain setting from a specific block. Services are synchronous and on-demand.
  • Updates: Information that the validator sends to a consensus engine, such as the arrival of a new block or receipt of a new consensus message from a peer. Updates are sent asynchronously as they occur.

Although you could use the API directly to implement a new consensus engine, the recommended interface is a consensus SDK. The SDK provides several useful classes that make it easier to implement a consensus engine. Sawtooth currently provides consensus SDKs for Python and Rust. We have used these SDKs to create the consensus engines for the PoET engine (Python), PBFT engine (Rust), and Raft engine (Rust).

These SDKs have a consistent design with an abstract Engine class, an engine Driver, and a validator Service. The abstract Engine class provides a clear starting point for new consensus engine implementations. If you plan to write your own consensus SDK, we recommend conforming to this design.

Try it Today!

One of the most important decisions for a distributed ledger application is the choice of consensus. By opening up this interface, we hope that each application built on Hyperledger Sawtooth can select the consensus algorithm that suits it best.

Hyperledger Sawtooth Blockchain Security (Part Three)

By | 网志, Hyperledger Sawtooth

This is the conclusion of my three-part series on Hyperledger Sawtooth Security. I started with Sawtooth consensus algorithms in part one, then continued with Sawtooth node and transaction processor security in part two. Here I will conclude by discussing Sawtooth application security and Sawtooth network security.

Client Application Security

The client part of a Sawtooth application is written by the application developer. The Sawtooth client communicates with a Sawtooth node by REST API requests, including signed transactions and batches. The signing is performed with a private key and, as such, key management and security is important. With Bitcoins, for example, poor key management has resulted in stolen Bitcoins and a “graveyard of Bitcoins” that are inaccessible forever. Key management is the responsibility of the client application as keys are not managed by Sawtooth software.

A keystore is where you securely store your keys. The public key for a keypair, used for signature verification, can be and should be distributed to anyone. The private key portion, used for signing, must be safeguarded from access by others. Here are some keystore methods, ordered from low to high security:

  • The minimum security used should restrict access to the private key. That is either restrict access to the machine holding the key or restrict read access to the private key file to the signer or (better yet) both
  • Better protection would be the use of software-encrypted keystore. This would be a private keystore accessible by a PIN
  • The best protection is from a Hardware Security Module (HSM) keystore or a network-accessible key manager, accessed using the Key Management Interoperability Protocol (KMIP)

Client Authentication

A Sawtooth client may take external user input. In which case, it is important to authenticate that the user is who they say they are. Authentication methods are usually categorized, from low to high security, into:

  • Single-factor Authentication (SFA). SFA is something you know. It could be something like a PIN, password, passphrase, or one-time password (OTP). The main disadvantage with SFA is it could be weak or hard to remember
  • Two-factor Authentication (2FA). 2FA is SFA plus something you have. It could be a security key, such as a U2F token (e.g., YubiKey). The main disadvantage with 2FA is it can be lost or stolen

  • Three-factor Authentication (3FA). 3FA is 1FA and 2FA plus something you are (biometrics). Examples include fingerprints, face recognition, or retina scan. The main disadvantages with 3FA is it can be forged and cannot be easily changed

With 2FA and 3FA, the idea is defense-in-depth (i.e., multiple hurdles to authenticate).

Network Security

Blockchains are subject to Distributed Denial of Service (DDoS) attacks. That is, an attack that attempts to overload blockchain nodes by flooding the targeted nodes with bogus messages. Classical public, unpermissioned blockchain networks avoid DDoS attacks because transactions require spending digital currency (such as Bitcoin), making attacks costly. Also, public blockchain networks are highly distributed—with thousands of nodes—making a DDoS attack on the entire network impractical.

Private or permissioned blockchains, such as Sawtooth, are not designed to run on a public network. As such, they do not require digital currency and “mining.”

Sawtooth network can and should be mitigated against DDoS attacks as follows:

  • Back pressure, a flow-control technique to reject unusually frequent client submissions. If the validator is overwhelmed, it will stop accepting new batches until it can handle more work. The number of batches the validator can accept is based on a multiplier (currently two) of a rolling average of the number of published batches.
  • Sawtooth communication uses the Zero Message Queue (ZMQ or 0MQ) message library. Sawtooth optionally enables encryption with ZMQ when the network_public_key and network_private_key settings are defined in validator.toml. For production, generate your own key pair instead of using a predefined key that may be present.
  • REST API input is validated to avoid buffer corruption or overflow attacks.
  • TCP port 4004, used for communication between internal validator node components, should be closed to outside access in any firewall configuration,
  • TCP port 5050, used to communicate between the validator node and the consensus engine, should be closed to outside access in any firewall configuration.
  • TCP port 8008, used for the REST API, should be closed to outside access in a firewall configuration providing all application clients accessing the REST API come from the local host.
  • If you use the Seth TP (for WASM smart contracts), TCP port 3030, used for Seth RPC, should be closed to outside access in a firewall configuration, providing all RPC requests come from the local host.
  • TCP port 8800, used to communicate between validator nodes, must be open to outside access in any firewall configuration.

Sawtooth validator nodes should be deployed on a VPN or other private network to prevent any outside access to Sawtooth TCP ports.

Basically, best practices dictate closing as many network ports as possible, encrypting network communications, and deploying in a protected network environment (such as a VPN).

Further Information

Announcing Hyperledger Sawtooth 1.1

By | 网志, Hyperledger Sawtooth

It is with great excitement that we would like to announce the release of Sawtooth version 1.1. Earlier this year we released Sawtooth 1.0, marking the production ready status of the platform. Since then the community has been hard at work adding new features, improving the privacy and performance of the platform, and growing the ecosystem.

The Sawtooth development team has been focused on two major new features for the Sawtooth 1.1 release, an improved consensus interface and support for WebAssembly smart contracts. For a full list of new features and improvements see the Sawtooth 1.1 Release Notes.

Improved consensus interface and new consensus options

While Sawtooth has always enabled ‘pluggable’ consensus and multiple consensus algorithms, recent experiences indicated that the existing consensus interface could be improved. Sawtooth has always aspired to be a modular platform that would enable lean experimentation and rapid adoption of new technologies, in particular, with regards to consensus. After analyzing a number of consensus algorithms that are available today, both Nakamoto (PoW/PoET) and classical (Raft/PBFT), the team decided to re-architect the consensus interface to improve the ease of integration. As a result of this new interface, the team has been able to port the existing Sawtooth consensus options, as well as add two new classical consensus options. Below is the state of these consensus options today:

    • Developer Mode (stable)
    • PoET-Simulator (Crash Fault Tolerant) (stable)
    • PoET-SGX (under development)
    • Raft (alpha)
    • PBFT (under development)

If you are interested in learning more about the new consensus interface, or writing your own, please see the detailed documentation.

Support for WebAssembly smart contracts (Sawtooth Sabre)

Sawtooth Sabre is a new smart contract engine for Sawtooth that enables the execution of WebAssembly-based smart contracts. WebAssembly (WASM) is a new web standard developed at the W3C with participation from major corporations like Apple, Google, and Microsoft. The Sawtooth Sabre project leverages an existing open source WASM interpreter from the broader blockchain community. This on-chain interpreter enables developers to write their code in a variety of languages, compile it down to WebAssembly, and then deploy it directly to the Sawtooth blockchain.

In addition to new feature development, the Sawtooth developer team has continued research and development on improving the privacy and performance of the Sawtooth platform.

Privacy:

On the privacy front, a new Hyperledger Lab called ‘Private Data Objects (PDO)’ has been created. PDO enables smart contracts to execute off-chain with confidentiality and integrity through the use of trusted execution environments. For more information, take a look at this video or read the paper. Private data objects are just one way of addressing blockchain confidentiality, but expect to see more techniques available to Sawtooth over the coming months.

Performance:

On the performance front, much of the effort has been spent porting core Sawtooth components from Python to Rust. While Python was a great language to start with, and enabled the team to rapidly iterate and define the appropriate modularity in the architecture, it is not the most performant language. The 1.0 release stabilized many of the Sawtooth APIs, and as we began tuning the system, we identified bottlenecks arising from the design of the Python programing language. The speed and type safety of the Rust programming language made it a natural fit for the evolution of Sawtooth. As of today, roughly 40% of the Sawtooth validator components have been ported to Rust, a number that we expect will continue to increase over time.

Finally, in addition to adding new features and improving the robustness of the Sawtooth platform, we have also seen an explosion of activity in the community, with dozens of new developers and a variety of tools and applications being openly built on top the Sawtooth infrastructure. Notable new projects in the Sawtooth ecosystem include:

Applications:

  • Sawtooth Supply Chain – A platform focused on supply train traceability with contributors from Bitwise IO and Cargill.
  • Sawtooth Next-Directory – An application focused on role-based access control with contributors from T-Mobile.

Tools:

  • Truffle integration with Sawtooth-Seth – A new integration that allows you to deploy Ethereum smart contracts to Sawtooth using the leading Ethereum development tool, Truffle. Built in collaboration with the Truffle team.
  • Caliper support for Sawtooth – Benchmark Sawtooth in a variety of configurations with Hyperledger Caliper.
  • Sawooth Explorer – A blockchain explorer built for Sawtooth by the team at PokitDok.
  • Grafana monitoring – A set of tools for data collection and visualization for live Sawtooth deployments.

Part of a Grafana dashboard for a Sawtooth Testnet running Raft

The Sawtooth ecosystem and functionality is rapidly expanding, which wouldn’t be possible without the community behind it. I’d like to thank all of the developers who have put in time building tools and applications, or providing support, for their effort, including, but not limited to:

Adam Gering, Adam Ludvik, Adam Parker, Al Hulaton, Amol Kulkarni, Andrea Gunderson, Andrew Backer, Andrew Donald Kennedy, Anne Chenette, Arthur Greef, Ashish Kumar Mishra, Benoit Razet, Boyd Johnson, Bridger Herman, Chris Spanton, Dan Anderson, Dan Middleton, Darian Plumb, Eloá Franca Verona, Gini Harrison, Griffin Howlett, James Mitchell, Joel Dudley, Jonathan Langlois, Kelly Olson, Keith Bloomfield Kenneth Koski, Kevin O’Donnell, Kevin Solorio, Logan Seeley, Manoj Gopalakrishnan, Michael Nguyen, Mike Zaccardo, Nick Drozd, Pankaj Goyal, PGobz, Patrick BUI, Peter Schwarz, Rajeev Ranjan, Richard Berg, Ry Jones, Ryan Banks, Ryan Beck-Buysse, Serge Koba, Shawn T. Amundson, Sutrannu, Tom Barnes, Tomislav Markovski, Yunhang Chen, Zac Delventhal, devsatishm, feihujiang, joewright, kidrecursive, mithunshashidhara, and ruffsl.

If you’d like to join the community or learn more, you can find more information here:

Chat: #Sawtooth in Hyperledger RocketChat

Docs: Sawtooth 1.1 Documentation

Code: Sawtooth-core Github

Website: Hyperledger Sawtooth Homepage

Thanks for reading and look forward to more posts detailing new Sawtooth 1.1 features and improvements. We encourage developers to try these new feature out and give us feedback!

 

Welcome Hyperledger Ursa!

By | 网志, Hyperledger Ursa

Hyperledger Ursa is the latest project to be accepted by the TSC! It is a modular, flexible cryptography library that is intended for—but not limited to—use by other projects in Hyperledger. Ursa’s objective is to make it much safer and easier for our distributed ledger projects to use existing, time tested, and trusted cryptographic libraries but also new cryptographic library implementations being developed.

Ursa aims to include things like a comprehensive library of modular signatures and symmetric-key primitives built on top of existing implementations, so blockchain developers can choose and modify their cryptographic schemes with a simple configuration file change. Ursa will also have implementations of newer, fancier cryptography, including things like pairing-based signatures, threshold signatures, and aggregate signatures, and also zero-knowledge primitives like SNARKs.

Ursa will be written mostly in Rust, but will have interfaces in all of the different languages that are commonly used throughout Hyperledger.

Why Ursa?

As Hyperledger has matured, the individual projects within Hyperledger have started to find a need for sophisticated cryptographic implementations. Rather than have each project implement its own cryptographic protocols, it is much better to collaborate on a shared library. There are many reasons to do this, including the following:

  1. Avoiding duplication: Crypto implementations are notoriously difficult to get correct (particularly when side channels are taken into account) and often require a lot of work in order to achieve a high level of security.  The library allows projects to share crypto implementations, avoiding unnecessary duplication and extra work.
  2. Security: Having most (or all) of the crypto code in a single location substantially simplifies the security analysis of the crypto portion of Hyperledger.  In addition, the lack of duplication means maintenance is easier (and thus, hopefully security bugs are less numerous). The presence of easy to use, secure crypto implementations might also make it less likely for less experienced people to create their own less secure implementations.  
  3. Expert Review: In addition, the ability to enforce expert review of all cryptographic code should increase security as well.  Having all of our cyptographic code in a single location makes it easier to concentrate all of the cryptographic expertise in the project and ensures that code will be well reviewed, thus decreasing the likelihood of dangerous security bugs.  
  4. Cross-platform interoperability: If two projects use the same crypto libraries, it simplifies (substantially in some cases) cross-platform interoperability, since cryptographic verification involves the same protocols on both sides.
  5. Modularity: This could be the first common component/module and a step towards modular DLT platforms, which share common components.   While we have already outlined most of the advantages this modularity brings in terms of actual functionality, a successful crypto library encourages and pushes forward more modular activities.
  6. New Projects: It is easier for new projects to get off the ground if they have easy access to well-implemented, modular cryptographic abstractions.

Who Is Involved in Ursa?

On the more practical side, Ursa currently includes developers who work on the security aspects of Hyperledger Indy, Sawtooth, and Fabric. In addition, the Ursa project includes several cryptographers with an academic background in theoretical cryptography to ensure that all cryptographic algorithms meet the desired levels of security.

Our goal in creating Ursa is to combine the efforts of all the security and cryptography experts in the Hyperledger community and move all of the projects forward.

Features and Plans

Currently Ursa has two distinct modules: a library for modular, flexible, and standardized basic cryptographic algorithms, and a library for more exotic cryptography, including so-called “smart” signatures and zero knowledge primitives called zmix.

Our first library is our “base crypto” library. Right now we are focused on our shared modular signature library, but we plan to extend this to allow easy modularization of all commonly used cryptographic primitives in Minicrypt. This—work in progress—has the implementation of several different signature schemes with a common API, allowing for blockchain builders to change signature schemes almost on-the-fly—or to use and support multiple signature schemes easily. Exact implementations and APIs have not been finalized, but they are in progress.

We note that there aren’t raw crypto implementations in this library—things here are stable and generally standardized—but wrappers for code from existing libraries and also code generated by commonly used cryptography libraries such as the Apache Milagro Crypto Library (AMCL). The novelty here is the modularization and API, which enables blockchain platforms to easily use a wide variety of changeable cryptographic algorithms without having to understand or interact with the underlying mathematics.

In the future, we expect other wrappings and modular code to go in this library. For instance, Indy makes use of aggregate signatures, a feature which the other platforms would also like available to them. There are also a variety of hash algorithms which provide different performance characteristics or support different signature schemes. Selecting vetted implementations and providing a common interface helps the Hyperledger community manage a growing crypto feature set in a responsible manner.

Our second initial subproject is zmix, which offers a generic way to create zero knowledge proofs that prove statements about multiple cryptographic building blocks, including signatures, commitments, and verifiable encryption. The goal of zmix is to provide a single flexible and secure implementation to construct such zero knowledge proofs. Zmix consists of C-callable code but there are also convenience wrappers for various programming languages.

Getting involved

If you’re interested in learning more about, using, or contributing to Ursa, please check out the following: https://www.hyperledger.org/projects/ursa

We welcome interest even from those who aren’t working with Hyperledger projects, so feel free to join us if you like!

All Are Welcome Here

By | 网志, Hyperledger Burrow, Hyperledger Fabric, Hyperledger Indy, Hyperledger Iroha, Hyperledger Sawtooth

A Minneapolis coffee shop that has fueled or at least caffeinated a lot of Hyperledger commits.

One of the first things people learn when coming to Hyperledger is that Hyperledger isn’t, like it’s name may imply, a ledger. It is a collection of blockchain technology projects. When we started out it was clear almost immediately that a single project could not satisfy the broad range of uses nor explore enough creative and useful approaches to fit those needs. Having a portfolio of projects, though, enables us to have the variety of ideas and contributors to become a strong open source community. Back in January of 2016 Sawtooth and Fabric were both on the horizon followed shortly by Iroha, but we wouldn’t have predicted that we would have Hyperledger Burrow and Hyperledger Indy – two projects that bear no resemblance to each other. Burrow is a permissioned Ethereum-based platform and Indy is a distributed identity ledger. Burrow is written in Go, and Indy was created in Python and is porting to Rust.

Both of these platforms are interesting in their own rights, but Hyperledger is even more interesting for the combination of these projects with the others. Both Sawtooth and Fabric have already integrated with Burrow’s EVM. Now Hyperledger has a set of offerings that can simultaneously satisfy diverse requirements for smart contract language, permissioning, and consensus. Likewise Sawtooth and Indy have been working together at our last several hackfests. The results of that may unlock new use cases and deployment architectures for distributed identity. So it’s not that our multiplicity of projects has given us strength through numbers, but rather strength through diversity.

Hyperledger Hackfest – December 2017 at The Underground Lisboa

The hackfests that we mentioned are one of the rare times that we get together face to face. Most of our collaboration is over mail list, chat, and pull-requests. When we do get together though it’s always in a new city with new faces. One of our most recent projects was hatched inside one of those buses. It wasn’t the most ergonomic meeting I’ve ever had but there was room for everyone on that bus.

Hyperledger Hackfest in Chicago

Our hackfest in Chicago was in a lot more conventional surroundings (still a very cool shared creative space .. lots of lab equipment and benches out of view on the other side of the wall to the right). Looking back at this photo is fun for me. I can see a lot of separate conversations happening at each table… people sharing different ideas, helping ramp new contributors, working on advancing new concepts with existing contributors. I can see a lot of similarity but also a little variety. It’s a busy room but there’s still open chairs and room for more variety.

Our next hackfest won’t be until March 2019 (Hyperledger is hosting Hyperledger Global Forum in December in Basel though). The March hackfest will be somewhere in Asia – location to be settled soon. The dates and locations of the other 2019 hackfests aren’t set yet. I don’t know where they will be specifically, but I do know that there will be a seat available and you will be welcome there.

These face to face meetings really are more the exception than the rule at Hyperledger. There are now more than 780 contributors spread all across the globe. 165 of those were just in the last few months. That means that every day we have a new person contributing to Hyperledger. Most of our engagement is through the development process. People contribute bug fixes, write new documentation, develop new features, file bugs, etc. If you’ve never contributed open source code before getting started might be intimidating. We don’t want it to be, though. There are a number of resources to help you get started. You can watch this quick video from Community Architect, Tracy Kuhrt. There’s documentation for each project, mail lists, a chat server, working groups, and some of the projects even host weekly phone calls to help new developers get engaged. Everyone in Hyperledger abides by a Code of Conduct so you can feel comfortable knowing that when you join any of those forums you will be treated respectfully. Anyone who wants to get involved can regardless of “physical appearance, race, ethnic origin, genetic differences, national or social origin, name, religion, gender, sexual orientation, family or health situation, pregnancy, disability, age, education, wealth, domicile, political view, morals, employment, or union activity.” We know that to get the best ideas, best code, best user experience we need your involvement. Please come join our community.

Image created by https://allarewelcomehere.us/ for Hyperledger

As always, you can keep up with what’s new with Hyperledger on Twitter or email us with any questions: info@hyperledger.org.