Welcome, Guest

Transferring large data
(1 viewing) (1) Guest
Go to bottomPage: 1
TOPIC: Transferring large data
#6
Transferring large data 10 Months, 4 Weeks ago  
Hi, I noticed an open bug report saying that when sending messages larger than 100kB, there is data loss. I need to be sending things like models, map data, point clouds, and they could be megabytes in size. What are my options? Should I just not use JAUS at all for this data and use something else (perhaps an ARQ communication mechanism)?
kentd
Go Gators!
Admin
Posts: 61
graphgraph
User Offline Click here to see the profile of this user
Gender: Male Draco098 Lights Out Photography Draco098 Draco098 Draco098 Location: Charlotte, NC Birthday: 09/20
The administrator has disabled public write access.
There's 10 types of people in the world; those that understand binary and those that don't.
 
#7
Re:Transferring large data 10 Months, 4 Weeks ago  
Well... the data loss is not a guarantee, I have sent very large messages in the past without issue. However, there is an issue where the send/recv buffer on the UDP loopback can drop packets if they are sent too quickly. Due to the nature of large message support in JAUS, this can occur after the message is chopped up into smaller chunks. There are a few ways to solve this...

  1. Intelligently throttle the sending of large message chunks so as to not overrun the loopback buffers.

  2. Unintelligently hack a sleep into the largeMessageHandler to ensure the buffer has enough time to be processed and not over run.

  3. Fundamentally change the interface between the NodeManager and components so that this cannot occur (shared memory or TCP are options).

  4. Don't use JAUS to transfer large chunks of data, use another protocol like FTP or SCP which is designed for large data transfer and just use JAUS to negotiate that exchange.


As you can see, each solution offers differing levels of complexity and robustness. Some approaches may break interoperability with existing OpenJAUS code and some may break interoperability with other JAUS tool kits. Each of these approaches are on the table for development right now, but we are currently debating and discussing the solution we want to ultimately deploy (and in fact I suspect it may be a combination of things).

All that said, you can certainly take any of the approaches and make something that just works. If you need something that fixes this today, I would recommend number 2. One issue here though is that a call into the largeMessageHandler belongs to the orginating thread and sleeps in there may cause lag in your state machine. So you may need to create an outbound queue, make a thread and process those messages, then kill the thread when the process is complete.

~Danny
kentd
Go Gators!
Admin
Posts: 61
graphgraph
User Offline Click here to see the profile of this user
Gender: Male Draco098 Lights Out Photography Draco098 Draco098 Draco098 Location: Charlotte, NC Birthday: 09/20
The administrator has disabled public write access.
There's 10 types of people in the world; those that understand binary and those that don't.
 
#8
Re:Transferring large data 10 Months, 4 Weeks ago  
Another very good approach that I have seen used successfully in the past:
Design your large data set messages such that you only send one small piece of the data at a time. Attach to each piece, a value of some kind that instructs the receiving side how / where to put that piece into the large data sent when reassembling it. Then send each piece of data as its own JAUS message.
kentd
Go Gators!
Admin
Posts: 61
graphgraph
User Offline Click here to see the profile of this user
Gender: Male Draco098 Lights Out Photography Draco098 Draco098 Draco098 Location: Charlotte, NC Birthday: 09/20
Last Edit: 2009/10/13 09:21 By kentd.
The administrator has disabled public write access.
There's 10 types of people in the world; those that understand binary and those that don't.
 
#9
Re:Transferring large data 10 Months, 4 Weeks ago  
I'm trying to use the PayloadDataElementMessage? messages to send a blob of data. I've been fiddling with:

-Creating a JausArray?, mallocing the dataElement to be the desired size, memcpy the data I want into it.
-jausPayloadInterface->jausInformationInterfaces = my array
-I'm not sure what to set jausPayloadInterface->jausCommandInterfaces to, but i've just created a JausArray? and set it to that.
-reportPayloadDataElement->jausPayloadInterface = jausPayloadInterface;
-reportPayloadDataElementMessageToJausMessage(reportPayloadDataElement). This is where I get a seg fault.

So I'm probably doing a few things wrong, and I was hoping someone can give me an example of how to use ReportPayloadDataElementMessage?, even a hacky, non-Jaus standard compliant method would be good.
OR
Can someone help me make a new experimental message that just takes a blob.

Thanks in advance
gibbocool
kentd
Go Gators!
Admin
Posts: 61
graphgraph
User Offline Click here to see the profile of this user
Gender: Male Draco098 Lights Out Photography Draco098 Draco098 Draco098 Location: Charlotte, NC Birthday: 09/20
The administrator has disabled public write access.
There's 10 types of people in the world; those that understand binary and those that don't.
 
#10
Re:Transferring large data 10 Months, 4 Weeks ago  
Creating an experimental message is actually quite easy! Have you looked at the tutorial on it?
Creating your own Experimental JAUS Message with OpenJAUS

The basic structure of the message you need would have two data members. Probably an unsigned integer for the size of the data and then a char pointer for the data. Packing and unpacking would be simple. Read the unsigned integer, then pack or unpack that number of characters from the buffer. One word of caution... make sure you properly implement the dataSize() method to return the number of bytes in the BLOB.

Again, let me know if you need additional help.
kentd
Go Gators!
Admin
Posts: 61
graphgraph
User Offline Click here to see the profile of this user
Gender: Male Draco098 Lights Out Photography Draco098 Draco098 Draco098 Location: Charlotte, NC Birthday: 09/20
The administrator has disabled public write access.
There's 10 types of people in the world; those that understand binary and those that don't.
 
#28
Re:Transferring large data 10 Months, 3 Weeks ago  
gibbocool wrote:
I got the blob working, it took a fair bit of debugging to figure out I need to malloc memory for the blob in dataFromBuffer. Perhaps put that in the tutorial?
kentd
Go Gators!
Admin
Posts: 61
graphgraph
User Offline Click here to see the profile of this user
Gender: Male Draco098 Lights Out Photography Draco098 Draco098 Draco098 Location: Charlotte, NC Birthday: 09/20
The administrator has disabled public write access.
There's 10 types of people in the world; those that understand binary and those that don't.
 
Go to topPage: 1
Copyright © 2010 OpenJAUS. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.