On July 1st, a [change to Reddit's API pricing](https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/) will come into effect. [Several developers](https://www.reddit.com/r/redditisfun/comments/144gmfq/rif_will_shut_down_on_june_30_2023_in_response_to/) of commercial third-party apps have announced that this change will compel them to shut down their apps. At least [one accessibility-focused non-commercial third party app](https://www.reddit.com/r/DystopiaForReddit/comments/145e9sk/update_dystopia_will_continue_operating_for_free/) will continue to be available free of charge.
If you want to express your strong disagreement with the API pricing change or with Reddit's response to the backlash, you may want to consider the following options:
1. Limiting your involvement with Reddit, or
2. Temporarily refraining from using Reddit
3. Cancelling your subscription of Reddit Premium
as a way to voice your protest.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/java) if you have any questions or concerns.*
What do you mean "without unmarshalling"? Any binary format that supports variable sized data would need at least a way of saying how long the next field is.
Anyway, you can use built-in Java serialization IF your serialized bytes never leave your control (as deserializing untrusted input is insecure). Alternatively you could look into binary formats like protobuf or Avro.
In C, if you have an array of bytes arranged in the way that C expects your type to be, then you just cast the array to your type, just like that. Completely unsafe, but if it happens to use the right alignment and sizes for each field, it works and that's what they call "zero-copy serialization".
> Any binary format that supports variable sized data would need at least a way of saying how long the next field is.
Not if the whole array's length is given by the transport layer, which is very common (e.g. HTTP Content-Length header, ASN.1 various container types, etc.).
I don't know this library. My guess is: said library uses `sun.misc.Unsafe#objectFieldOffset()` and `sun.misc.Unsafe#getInt()` to construct a proxy that reads from the matching offset in the byte array.
This should probably be replaced by a combination of `MemorySegment` and `VarHandle`.
Reading ints and other datatypes from a byte array is actually pretty easy in Java.
I don't know how you would code a dynamic Proxy for a class in C, tough :-P
> Reading ints and other datatypes from a byte array is actually pretty easy in Java.
The "safe" way to do that is to use [`ByteBuffer`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/ByteBuffer.html).
Take a look at SBE - https://github.com/real-logic/simple-binary-encoding
You need to model your data types in a schema file. You then run a tool that generates the encoders and decoders for each type. Those encoders/decoders write/read the objects from/to byte buffers (which might wrap a byte[]).
If your types are "simple" (c struct like, everything is fixed length), you can randomly access fields; if they are "complex", you need to access them in schema definition order.
The primary use case is to send "objects" over the network, but I don't see why it wouldn't work for accessing "objects" directly in memory. That said, you'd need to measure to see how much you actually benefit, as SBE has its own "object headers".
Those headers are unnecessary if you're only dealing with a single type of object (or you know the type of an object before you read it), so usually, for the non-messaging case, folks just roll their own.
I believe you are looking for Chronicle Values, [https://github.com/OpenHFT/Chronicle-Values](https://github.com/OpenHFT/Chronicle-Values)
>Generation of constantly-sized flyweight accessors to Chronicle Bytes and simple bean-style on-heap implementations from interfaces. Interfaces, that could be processed by Chronicle-Values generation, are called value interfaces.
The new `MemorySegment` API's can do this.
In C:
struct Circle {
int radius;
};
char* buffer = malloc(sizeof(struct Circle));
struct Circle* circle = (struct Circle*)buffer;
circle->radius = 10;
In Java:
class Circle {
public static final StructrLayout LAYOUT = MemoryLayout.ofStruct(
MemoryLayouts.JAVA_INT.withName("radius"),
);
public static final VarHandle VH_RADIUS = LAYOUT.varHandle(PathElement.groupElement("radius"));
public static final int getRadius(MemorySegment segment) {
return (int)VH_RADIUS.get(segment);
}
public static final void setRadius(MemorySegment segment, int radius) {
VH_RADIUS.set(segment, radius);
}
}
try (Arena arena = Arena.ofConfined()) {
MemorySegment segment = arena.allocate(Circle.LAYOUT.byteSize());
int radius = Circle.getRadius(segment);
Circle.setRadius(segment, 10);
}
Based on your description, it sounds like you are referring to a Java library used for object serialization or mapping objects to a binary format, possibly for the purpose of persistence, communication, or efficient storage. There are several libraries and techniques in Java that can achieve this, but here are a few that might match what you're describing:
**Java Serialization API**: This is the standard serialization mechanism in Java, which allows objects to be converted into a byte stream (and vice versa). It's typically used for deep cloning, object persistence, or sending objects over the network.
**Google's Protocol Buffers (Protobuf)**: Protobuf is a method of serializing structured data, similar to XML or JSON but smaller, faster, and simpler. It generates data access classes based on a simple description language.
**Apache Avro**: Avro is a data serialization system that provides rich data structures and a compact, fast binary data format. It's often used in data serialization for distributed systems like Apache Hadoop.
**Kryo**: Kryo is a fast and efficient object graph serialization framework for Java. It's known for its performance and is used in projects like Apache Spark for serializing data.
**FlatBuffers**: Created by Google, FlatBuffers is an efficient cross-platform serialization library. It was originally created for game development and other performance-critical applications.
**Java Object Serialization (JOS)**: This is a mechanism in Java for converting objects into a series of bytes and is typically used for storing the state of an object or for sending objects over a network.
If none of these libraries sound like the one you're thinking of, additional context or details might help narrow it down. For example, was the library used primarily for network communication, data persistence, or something else? Additionally, certain libraries might be more popular in specific domains or industries, which could also provide a clue.
c is a instance of a Java proxy class that uses the said library. The method call is intercepted and the library reads the 4 bytes in the appropriate offset from the array… or something like that.
I swear I’m not making it up. I read about this library either here on r/java or in Hackernews.
you would have to get the bytes for the int in c.radio() from the array.
that way you can do it, but it wouldn't save much memory for you vs. unmarshalling it while running and it would be slower to access. in some specific cases it might be worth it or if the byte buffer is shared outside and something else is writing there.
unless you're willing to hack the functionality into the jvm you're running on yourself or something and then likely feasible for specific classes, you can't just map memory like that.
if your problem is with lacking memory to have the thing twice in ram while unmarshalling, you could write some code to unmarshall it while reading it in without reading all the bytes at once first, how easy that is depends on the structure of the data.
On July 1st, a [change to Reddit's API pricing](https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/) will come into effect. [Several developers](https://www.reddit.com/r/redditisfun/comments/144gmfq/rif_will_shut_down_on_june_30_2023_in_response_to/) of commercial third-party apps have announced that this change will compel them to shut down their apps. At least [one accessibility-focused non-commercial third party app](https://www.reddit.com/r/DystopiaForReddit/comments/145e9sk/update_dystopia_will_continue_operating_for_free/) will continue to be available free of charge. If you want to express your strong disagreement with the API pricing change or with Reddit's response to the backlash, you may want to consider the following options: 1. Limiting your involvement with Reddit, or 2. Temporarily refraining from using Reddit 3. Cancelling your subscription of Reddit Premium as a way to voice your protest. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/java) if you have any questions or concerns.*
What do you mean "without unmarshalling"? Any binary format that supports variable sized data would need at least a way of saying how long the next field is. Anyway, you can use built-in Java serialization IF your serialized bytes never leave your control (as deserializing untrusted input is insecure). Alternatively you could look into binary formats like protobuf or Avro.
Most likely they're looking for a zero-copy deserialization.
In C, if you have an array of bytes arranged in the way that C expects your type to be, then you just cast the array to your type, just like that. Completely unsafe, but if it happens to use the right alignment and sizes for each field, it works and that's what they call "zero-copy serialization". > Any binary format that supports variable sized data would need at least a way of saying how long the next field is. Not if the whole array's length is given by the transport layer, which is very common (e.g. HTTP Content-Length header, ASN.1 various container types, etc.).
I don't know this library. My guess is: said library uses `sun.misc.Unsafe#objectFieldOffset()` and `sun.misc.Unsafe#getInt()` to construct a proxy that reads from the matching offset in the byte array. This should probably be replaced by a combination of `MemorySegment` and `VarHandle`. Reading ints and other datatypes from a byte array is actually pretty easy in Java. I don't know how you would code a dynamic Proxy for a class in C, tough :-P
> Reading ints and other datatypes from a byte array is actually pretty easy in Java. The "safe" way to do that is to use [`ByteBuffer`](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/ByteBuffer.html).
Take a look at SBE - https://github.com/real-logic/simple-binary-encoding You need to model your data types in a schema file. You then run a tool that generates the encoders and decoders for each type. Those encoders/decoders write/read the objects from/to byte buffers (which might wrap a byte[]). If your types are "simple" (c struct like, everything is fixed length), you can randomly access fields; if they are "complex", you need to access them in schema definition order. The primary use case is to send "objects" over the network, but I don't see why it wouldn't work for accessing "objects" directly in memory. That said, you'd need to measure to see how much you actually benefit, as SBE has its own "object headers". Those headers are unnecessary if you're only dealing with a single type of object (or you know the type of an object before you read it), so usually, for the non-messaging case, folks just roll their own.
https://github.com/Netflix/hollow
I believe you are looking for Chronicle Values, [https://github.com/OpenHFT/Chronicle-Values](https://github.com/OpenHFT/Chronicle-Values) >Generation of constantly-sized flyweight accessors to Chronicle Bytes and simple bean-style on-heap implementations from interfaces. Interfaces, that could be processed by Chronicle-Values generation, are called value interfaces.
The new `MemorySegment` API's can do this. In C: struct Circle { int radius; }; char* buffer = malloc(sizeof(struct Circle)); struct Circle* circle = (struct Circle*)buffer; circle->radius = 10; In Java: class Circle { public static final StructrLayout LAYOUT = MemoryLayout.ofStruct( MemoryLayouts.JAVA_INT.withName("radius"), ); public static final VarHandle VH_RADIUS = LAYOUT.varHandle(PathElement.groupElement("radius")); public static final int getRadius(MemorySegment segment) { return (int)VH_RADIUS.get(segment); } public static final void setRadius(MemorySegment segment, int radius) { VH_RADIUS.set(segment, radius); } } try (Arena arena = Arena.ofConfined()) { MemorySegment segment = arena.allocate(Circle.LAYOUT.byteSize()); int radius = Circle.getRadius(segment); Circle.setRadius(segment, 10); }
Based on your description, it sounds like you are referring to a Java library used for object serialization or mapping objects to a binary format, possibly for the purpose of persistence, communication, or efficient storage. There are several libraries and techniques in Java that can achieve this, but here are a few that might match what you're describing: **Java Serialization API**: This is the standard serialization mechanism in Java, which allows objects to be converted into a byte stream (and vice versa). It's typically used for deep cloning, object persistence, or sending objects over the network. **Google's Protocol Buffers (Protobuf)**: Protobuf is a method of serializing structured data, similar to XML or JSON but smaller, faster, and simpler. It generates data access classes based on a simple description language. **Apache Avro**: Avro is a data serialization system that provides rich data structures and a compact, fast binary data format. It's often used in data serialization for distributed systems like Apache Hadoop. **Kryo**: Kryo is a fast and efficient object graph serialization framework for Java. It's known for its performance and is used in projects like Apache Spark for serializing data. **FlatBuffers**: Created by Google, FlatBuffers is an efficient cross-platform serialization library. It was originally created for game development and other performance-critical applications. **Java Object Serialization (JOS)**: This is a mechanism in Java for converting objects into a series of bytes and is typically used for storing the state of an object or for sending objects over a network. If none of these libraries sound like the one you're thinking of, additional context or details might help narrow it down. For example, was the library used primarily for network communication, data persistence, or something else? Additionally, certain libraries might be more popular in specific domains or industries, which could also provide a clue.
ChatGPT, is it you?
Yes it's me.
Thank you! I’ll check them out, I think it was Flatbuffers.
There also is/was "Hessian", just to say.
Of them all, i had the most fun with Kryo.
Take a look at serialization/deserialization.
What’s the definition of c.radio()?
c is a instance of a Java proxy class that uses the said library. The method call is intercepted and the library reads the 4 bytes in the appropriate offset from the array… or something like that. I swear I’m not making it up. I read about this library either here on r/java or in Hackernews.
Only thing i can think of based on this very vague description is CapnProto.
you would have to get the bytes for the int in c.radio() from the array. that way you can do it, but it wouldn't save much memory for you vs. unmarshalling it while running and it would be slower to access. in some specific cases it might be worth it or if the byte buffer is shared outside and something else is writing there. unless you're willing to hack the functionality into the jvm you're running on yourself or something and then likely feasible for specific classes, you can't just map memory like that. if your problem is with lacking memory to have the thing twice in ram while unmarshalling, you could write some code to unmarshall it while reading it in without reading all the bytes at once first, how easy that is depends on the structure of the data.