Fast parallel async HTTP/SSH/TCP/UDP/Ping client java library on Akka. Aggregate 100,000 APIs & send results anywhere in 20 lines of code. View production use cases.
Ping or HTTP calls 8000 servers with responses aggregated in 12 seconds. Parallec means Parallel Client (pronounced as "para-like"). Open Source from eBay Cloud.
Scalable Web server management
Asset discovery & remote task execution
API aggregation
Orchestration
Data Extraction / ETL
Load testing
Deployed in Production 8,000 Server HTTP | Ping Demo
Tweeted by the Creator of Akka
Featured in This Week in #Scala | OSChina - 2015 Top 100
Aggregated error messages - Debug friendly with full visibility. Have trouble debugging in concurrent environment? Not any more! All exceptions, timeout, stack traces, request sent and response received time are automatically captured and aggregated in the response map of the ParallelTask, available for polling asynchronously.
A convenient response context passes any object you need when handling response. Process data anyway and send it anywhere.
Read MoreIntuitive builder pattern APIs make parallel requests exceedlingly easy. Input target hosts from list, string, text, json path, cms query from local or urls.
Read MoreAuto progress polling to handle async APIs such as "download packages" or "create compute", enabling task level concurrency control and orchestration.
Read MoreInfinite scalablility with flexible task level concurrency control without creating 1,000 threads thread pools. Ping / HTTP 8,000 servers with response aggregation in 12 seconds.
Read MoreHigh test coverage validates every feature we built. Use and contribute with confidence. Proven reliability on hitting 100,000+ Servers in a single task in production.
Check CoverageTask and host level progress tracking and cancelation, capacity-aware task scheduler, flexible handler locations, status aggregation, task logging, and more.
Read MoreMonitoring, config push, command execution, discovery by polling from massive HTTP/TCP agents, or agent-less parallel SSH. Perfect for embedding in the master to manage 100,000s of end points, in 20 lines of code.
Learn more: why such a polling and aggregation engine is widely used in today’s cloud software.
A special super convenient response context let you pass in/out any object when handling the response. Now you can send aggregated data anywhere, to Kafka, Elastic Search, Redis, MongoDb, and etc, in 20 lines of code. More Features...
6 Lines Example to Get Started.
import io.parallec.core.*;
import java.util.Map;
ParallelClient pc = new ParallelClient();
pc.prepareHttpGet("").setTargetHostsFromString("www.google.com www.ebay.com www.yahoo.com")
.execute(new ParallecResponseHandler() {
public void onCompleted(ResponseOnSingleTask res,
Map<String, Object> responseContext) {
System.out.println( res.toString() ); }
});
Different Requests to the Same Target Example. Read more..
pc.prepareHttpGet("/userdata/sample_weather_$ZIP.txt")
.setReplaceVarMapToSingleTargetSingleVar("ZIP",
Arrays.asList("95037","48824"), "www.parallec.io")
.setConcurrency(10)
.execute(new ParallecResponseHandler() {...}...
20 Lines Example using Response Context: Aggregate 100 website status to elastic search.
ParallelClient pc = new ParallelClient();
org.elasticsearch.node.Node node = nodeBuilder().node(); //elastic client initialize
HashMap<String, Object> responseContext = new HashMap<String, Object>();
responseContext.put("Client", node.client());
pc.prepareHttpGet("")
.setConcurrency(1000).setResponseContext(responseContext)
.setTargetHostsFromLineByLineText("http://www.parallec.io/userdata/sample_target_hosts_top100_old.txt", HostsSourceType.URL)
.execute( new ParallecResponseHandler() {
public void onCompleted(ResponseOnSingleTask res,
Map<String, Object> responseContext) {
Map<String, Object> metricMap = new HashMap<String, Object>();
metricMap.put("StatusCode", res.getStatusCode().replaceAll(" ", "_"));
metricMap.put("LastUpdated",PcDateUtils.getNowDateTimeStrStandard());
metricMap.put("NodeGroupType", "Web100");
Client client = (Client) responseContext.get("Client");
client.prepareIndex("local", "parallec", res.getHost()).setSource(metricMap).execute();
}
});
node.close(); pc.releaseExternalResources();
<dependency>
<groupId>io.parallec</groupId>
<artifactId>parallec-core</artifactId>
<version>0.10.3</version>
</dependency>
compile 'io.parallec:parallec-core:0.10.3'
More Examples and Complete Code on sending to Kafka, async running, parallel SSH and TCP.
Set Target Hosts from list, string, line by line text, json path, from local or remote URLs.
Compared with java thread pool based solutions, parallec enables adjustable concurrency control on each task without constraints on thread size (1,000 thread pool does not look nice) . Compared with single-threaded Node.js solutions, Parallec enables parallel computation-intensive response handling with multiple-cores. Read More
Our goal is to execute a task of firing a list of uniform or non-uniform HTTP/SSH/TCP/UDP/PING requests to a list of target hosts, then call the user defined handler to handle each of the responses. Check Parallec's Actor based concurrency and throttling model.
Most files are independent with a main function and can be run directly. The sample spark server is a single file executable web server, which demonstrates parallel SSH/HTTP/Ping features.