Using a Task to Periodically Pull in External Data
A Task is an entity that can periodically pull data from external systems and "inject" it into Octave as events. A Task consists of a destination stream where the events are to be placed in Octave, periodicity which specifies how often to poll for external data, and a source which specifies where to pull the data from.
The source type can be set to http to read data from a URL, or eventFind to read from a specified a streamid or streamPath.
When using http, a parser type (e.g. RSS, JSON, etc.) must be specified so Octave knows how to read/extract the source data.
eventFind can be used for finding, sorting, and/ordering events. For example, it can be used in a Cloud Action to query a stream of light readings created by a Cloud Action every five minutes, report on bad devices, and push data to an external service such as a dashboard.
Tasks are created and managed by issuing the following REST requests from an external system:
Creating a Task
A new task is created by invoking the POST /task endpoint. In the following example, a Task is created that reads data from an RSS feed everything 20 seconds and writes to the /task_output stream. Octave's rss parser is used to extract the data and convert it into Stream events:
Request
## Create Task
curl -X "POST" "<%= config[:base_api_url] %><%= config[:sample_company] %>/task" \
-H 'X-Auth-Token: <token>' \
-H 'X-Auth-User: <user>' \
-H 'Content-Type: application/json; charset=utf-8' \
-d $'{
"destination": "/<%= config[:sample_company] %>/task_output",
"periodicity": "20000",
"source": {
"http": {
"get": {
"url": "https://www.nasa.gov/rss/dyn/breaking_news.rss"
},
"parser": "rss"
}
},
"displayName": "A sample task"
}'
Response
The messages field indicates the result of request and the body field provides details about the new task such as its unique id:
{
"head":{
"status":201,
"ok":true,
"messages":[
"Your request has been processed successfully. A new resource has been created."
],
"errors":[
],
"references":{
}
},
"body":{
"id":"t5d2df800c2b092ef7a3475d3",
"companyId":"c5adf803aaa02b35e0cce7b2d",
"creationDate":1563293696545,
"creatorId":"i5adf803aaa02b35e0cce7b2a",
"destination":"/<%= config[:sample_company] %>/task_output",
"displayName":"A sample task",
"lastRun":0,
"nextRun":0,
"periodicity":20000,
"runCt":0,
"source":{
"http":{
"parser":"rss",
"get":{
"url":"https://www.nasa.gov/rss/dyn/breaking_news.rss"
}
}
},
"status":"OK"
}
}
Reading a Task
Information about an existing Task can be obtained at any time by invoking GET /tasks/{id} and passing the ID of the Task as a path parameter:
curl "<%= config[:base_api_url] %><%= config[:sample_company] %>/task/t5d2df800c2b092ef7a3475d3" \
-H 'X-Auth-Token: <token>' \
-H 'X-Auth-User: <user>'
Response
The body field provides details about the task:
{
"head": {
"status": 200,
"ok": true,
"messages": [],
"errors": [],
"references": {}
},
"body": {
"id": "t5d2df800c2b092ef7a3475d3",
"companyId": "c5adf803aaa02b35e0cce7b2d",
"creationDate": 1563293696545,
"creatorId": "i5adf803aaa02b35e0cce7b2a",
"description": "A RSS news feed containing the latest NASA news articles and press releases.",
"destination": "/<%= config[:sample_company] %>/task_output",
"displayName": "A sample task",
"lastRun": 1563296216786,
"nextRun": 1563296236786,
"periodicity": 20000,
"runCt": 127,
"source": {
"http": {
"parser": "rss",
"get": {
"url": "https://www.nasa.gov/rss/dyn/breaking_news.rss"
}
}
},
"status": "NO_NEW_DATA"
}
}
Updating a Task
An existing Task can be updated by invoking the PUT /task/{endpoint} endpoint and passing the ID of the Task as a path parameter; the new Task property values are passed as body parameters. In the following example, the description property is updated to "My Task":
curl -X "PUT" "<%= config[:base_api_url] %><%= config[:sample_company] %>/task/t5d2df800c2b092ef7a3475d3" \
-H 'X-Auth-Token: <token>' \
-H 'X-Auth-User: <user>' \
-d $'{
"description": "My Task"
}'
Response
{
"head":{
"status":200,
"ok":true,
"messages":[
],
"errors":[
],
"references":{
}
},
"body":{
"id":"t5d2df800c2b092ef7a3475d3",
"companyId":"c5adf803aaa02b35e0cce7b2d",
"creationDate":1563293696545,
"creatorId":"i5adf803aaa02b35e0cce7b2a",
"description":"My Task",
"destination":"/<%= config[:sample_company] %>/task_output",
"displayName":"A sample task",
"lastEditDate":1563302760142,
"lastEditorId":"i5adf803aaa02b35e0cce7b2a",
"lastRun":1563302756772,
"nextRun":1563302776772,
"periodicity":20000,
"runCt":454,
"source":{
"http":{
"parser":"rss",
"get":{
"url":"https://www.nasa.gov/rss/dyn/breaking_news.rss"
}
}
},
"status":"OK"
}
}
Deleting a Task
An existing task can be deleted by invoking the DELETE /task/{id} endpoint and passing the ID of the Task as a path parameter:
curl -X "DELETE" "<%= config[:base_api_url] %><%= config[:sample_company] %>/task/t5d2df800c2b092ef7a3475d3" \
-H 'X-Auth-Token: <token>' \
-H 'X-Auth-User: <user>'
Response
The messsages field in the response indicates the result of the request:
{
"head":{
"status":200,
"ok":true,
"messages":[
"Your request has been processed successfully. The requested resource has been deleted."
],
"errors":[
],
"references":{
}
},
"body":{
}
}
Simulating a Task
A Task can be "simulated" to show what an event from a given source would look like, by invoking the POST /task/simulate endpoint. Similar to the Create Task endpoint, the simulate endpoint requires a source stream, periodicity, and destination stream.
curl -X "POST" "<%= config[:base_api_url] %><%= config[:sample_company] %>/task/simulate" \
-H 'X-Auth-Token: <token>' \
-H 'X-Auth-User: <user>' \
-H 'Content-Type: text/plain; charset=utf-8' \
-d $'{
"source": {
"http": {
"get": {
"url": "https://www.nasa.gov/rss/dyn/breaking_news.rss"
},
"parser": "rss"
}
},
"periodicity": "20000",
"destination": "/<%= config[:sample_company] %>/task_output",
"displayName": "A sample task"
}'
Response
The events field in the response provides a preview of the event JSON that would be generated by Octave from the source data:
{
"head":{
"status":200,
"ok":true,
"messages":[
],
"errors":[
],
"references":{
}
},
"body":{
"events":{
"/<%= config[:sample_company] %>/task_out":[
{
"metadata":null,
"creationDate":null,
"lastEditDate":null,
"generatedDate":null,
"path":null,
"location":null,
"hash":"http://www.nasa.gov/press-release/nasa-scientists-engineers-honored-with-presidential-early-career-awards",
"tags":null,
"elems":{
"thumbnail":{
"media_type":"image",
"mime_type":"image/jpeg",
"length":189751,
"url":"http://www.nasa.gov/sites/default/files/styles/1x1_cardfeed/public/thumbnails/image/nasa-logo-web-rgb_0.jpg?itok=mCRPbzgr"
},
"link":"http://www.nasa.gov/press-release/nasa-scientists-engineers-honored-with-presidential-early-career-awards",
"publication_date":1563305489105,
"description":"President Trump has named 18 NASA researchers as recipients of the Presidential Early Career Award for Scientists and Engineers (PECASE).",
"title":"NASA Scientists, Engineers Honored with Presidential Early Career Awards",
"content":{
"type":"html",
"value":"President Trump has named 18 NASA researchers as recipients of the Presidential Early Career Award for Scientists and Engineers (PECASE)."
}
}
}
]
}
}
}
Updated about 5 years ago