code cov   code cov   code cov   code cov

Parallec.io

Fast parallel async HTTP/SSH/TCP/UDP/Ping client java library on Akka. Aggregate 100,000 APIs & send results anywhere in 20 lines of code. View production use cases.
Ping or HTTP calls 8000 servers with responses aggregated in 12 seconds. Parallec means Parallel Client (pronounced as "para-like"). Open Source from eBay Cloud.

Scalable Web server management Asset discovery & remote task execution API aggregation Orchestration Data Extraction / ETL Load testing
Deployed in Production 8,000 Server HTTP | Ping Demo Tweeted by the Creator of Akka Featured in This Week in #Scala | OSChina - 2015 Top 100

Aggregated error messages - Debug friendly with full visibility. Have trouble debugging in concurrent environment? Not any more! All exceptions, timeout, stack traces, request sent and response received time are automatically captured and aggregated in the response map of the ParallelTask, available for polling asynchronously.


Response Handler & Context

A convenient response context passes any object you need when handling response. Process data anyway and send it anywhere.

Read More

Exceedingly User Friendly

Intuitive builder pattern APIs make parallel requests exceedlingly easy. Input target hosts from list, string, text, json path, cms query from local or urls.

Read More

Auto Progress Polling

Auto progress polling to handle async APIs such as "download packages" or "create compute", enabling task level concurrency control and orchestration.

Read More

Scalable & Fast

Infinite scalablility with flexible task level concurrency control without creating 1,000 threads thread pools. Ping / HTTP 8,000 servers with response aggregation in 12 seconds.

Read More

90%+ Code Coverage

High test coverage validates every feature we built. Use and contribute with confidence. Proven reliability on hitting 100,000+ Servers in a single task in production.

Check Coverage

Feature Packed

Task and host level progress tracking and cancelation, capacity-aware task scheduler, flexible handler locations, status aggregation, task logging, and more.

Read More

Try Parallec Now

Monitoring, config push, command execution, discovery by polling from massive HTTP/TCP agents, or agent-less parallel SSH. Perfect for embedding in the master to manage 100,000s of end points, in 20 lines of code.
Learn more: why such a polling and aggregation engine is widely used in today’s cloud software.

Send Aggregated API Data Anywhere

A special super convenient response context let you pass in/out any object when handling the response. Now you can send aggregated data anywhere, to Kafka, Elastic Search, Redis, MongoDb, and etc, in 20 lines of code. More Features...

6 Lines Example to Get Started.

import io.parallec.core.*;
import java.util.Map;

ParallelClient pc = new ParallelClient();
pc.prepareHttpGet("").setTargetHostsFromString("www.google.com www.ebay.com www.yahoo.com")
.execute(new ParallecResponseHandler() {
    public void onCompleted(ResponseOnSingleTask res,
        Map<String, Object> responseContext) {
        System.out.println( res.toString() );  }
});

Different Requests to the Same Target Example. Read more..

pc.prepareHttpGet("/userdata/sample_weather_$ZIP.txt")
    .setReplaceVarMapToSingleTargetSingleVar("ZIP",
        Arrays.asList("95037","48824"), "www.parallec.io")
    .setConcurrency(10)
    .execute(new ParallecResponseHandler() {...}...

20 Lines Example using Response Context: Aggregate 100 website status to elastic search.

ParallelClient pc = new ParallelClient();
org.elasticsearch.node.Node node = nodeBuilder().node(); //elastic client initialize
HashMap<String, Object> responseContext = new HashMap<String, Object>();
responseContext.put("Client", node.client());
pc.prepareHttpGet("")
        .setConcurrency(1000).setResponseContext(responseContext)
        .setTargetHostsFromLineByLineText("http://www.parallec.io/userdata/sample_target_hosts_top100_old.txt", HostsSourceType.URL)
        .execute( new ParallecResponseHandler() {
            public void onCompleted(ResponseOnSingleTask res,
                    Map<String, Object> responseContext) {
                Map<String, Object> metricMap = new HashMap<String, Object>();
                metricMap.put("StatusCode", res.getStatusCode().replaceAll(" ", "_"));
                metricMap.put("LastUpdated",PcDateUtils.getNowDateTimeStrStandard());
                metricMap.put("NodeGroupType", "Web100");
                Client client = (Client) responseContext.get("Client");
                client.prepareIndex("local", "parallec", res.getHost()).setSource(metricMap).execute();
            }
        });
node.close(); pc.releaseExternalResources();

Maven Import

<dependency>
	<groupId>io.parallec</groupId>
	<artifactId>parallec-core</artifactId>
	<version>0.10.3</version>
</dependency>

Gradle Import

compile 'io.parallec:parallec-core:0.10.3'

More Examples and Complete Code on sending to Kafka, async running, parallel SSH and TCP.

Set Target Hosts from list, string, line by line text, json path, from local or remote URLs.

Why Parallec ?

Compared with java thread pool based solutions, parallec enables adjustable concurrency control on each task without constraints on thread size (1,000 thread pool does not look nice) . Compared with single-threaded Node.js solutions, Parallec enables parallel computation-intensive response handling with multiple-cores. Read More

Workflow

Flow Overview

Our goal is to execute a task of firing a list of uniform or non-uniform HTTP/SSH/TCP/UDP/PING requests to a list of target hosts, then call the user defined handler to handle each of the responses. Check Parallec's Actor based concurrency and throttling model.

Read API Overview

Try with Sample Code

Most files are independent with a main function and can be run directly. The sample spark server is a single file executable web server, which demonstrates parallel SSH/HTTP/Ping features.