cudf::io::data_sink Class Reference

Interface class for storing the output data from the writers. More...

#include <data_sink.hpp>

Public Member Functions
virtual	~data_sink ()
	Base class destructor.
virtual void	host_write (void const *data, size_t size)=0
	Append the buffer content to the sink. More...
virtual bool	supports_device_write () const
	Whether or not this sink supports writing from gpu memory addresses. More...
virtual bool	is_device_write_preferred (size_t size) const
	Estimates whether a direct device write would be more optimal for the given size. More...
virtual void	device_write (void const *gpu_data, size_t size, rmm::cuda_stream_view stream)
	Append the buffer content to the sink from a gpu address. More...
virtual std::future< void >	device_write_async (void const *gpu_data, size_t size, rmm::cuda_stream_view stream)
	Asynchronously append the buffer content to the sink from a gpu address. More...
virtual void	flush ()=0
	Flush the data written into the sink.
virtual size_t	bytes_written ()=0
	Returns the total number of bytes written into this sink. More...

Static Public Member Functions
static std::unique_ptr< data_sink >	create (std::string const &filepath)
	Create a sink from a file path. More...
static std::unique_ptr< data_sink >	create (std::vector< char > *buffer)
	Create a sink from a std::vector. More...
static std::unique_ptr< data_sink >	create ()
	Create a void sink (one that does no actual io) More...
static std::unique_ptr< data_sink >	create (cudf::io::data_sink *const user_sink)
	Create a wrapped custom user data sink. More...
template<typename T >
static std::vector< std::unique_ptr< data_sink > >	create (std::vector< T > const &args)
	Creates a vector of data sinks, one per element in the input vector. More...

Detailed Description

Interface class for storing the output data from the writers.

Definition at line 43 of file data_sink.hpp.

Member Function Documentation

bytes_written()

virtual size_t cudf::io::data_sink::bytes_written ( )

pure virtual

Returns the total number of bytes written into this sink.

Returns

Total number of bytes written into this sink

create() [1/5]

static std::unique_ptr<data_sink> cudf::io::data_sink::create ( )

static

Create a void sink (one that does no actual io)

A useful code path for benchmarking, to eliminate physical hardware randomness from profiling.

Returns

Constructed data_sink object

create() [2/5]

static std::unique_ptr<data_sink> cudf::io::data_sink::create ( cudf::io::data_sink *const  user_sink )

static

Create a wrapped custom user data sink.

Parameters

[in]

user_sink

User-provided data sink (typically custom class)

The data sink returned here is not the one passed by the user. It is an internal class that wraps the user pointer. The principle is to allow the user to declare a custom sink instance and use it across multiple write() calls.

Returns

Constructed data_sink object

create() [3/5]

static std::unique_ptr<data_sink> cudf::io::data_sink::create ( std::string const &  filepath )

static

Create a sink from a file path.

Parameters

[in]

filepath

Path to the file to use

Returns

Constructed data_sink object

create() [4/5]

static std::unique_ptr<data_sink> cudf::io::data_sink::create ( std::vector< char > *  buffer )

static

Create a sink from a std::vector.

Parameters

[in,out]

buffer

Pointer to the output vector

Returns

Constructed data_sink object

create() [5/5]

template<typename T >

static std::vector<std::unique_ptr<data_sink> > cudf::io::data_sink::create ( std::vector< T > const &  args )

inlinestatic

Creates a vector of data sinks, one per element in the input vector.

Parameters

[in]

args

vector of parameters

Returns

Constructed vector of data sinks

Definition at line 91 of file data_sink.hpp.

device_write()

virtual void cudf::io::data_sink::device_write ( void const *  gpu_data, size_t  size, rmm::cuda_stream_view  stream  )

inlinevirtual

Append the buffer content to the sink from a gpu address.

For optimal performance, should only be called when is_device_write_preferred returns true. Data sink implementations that don't support direct device writes don't need to override this function.

Exceptions

cudf::logic_error

the object does not support direct device writes, i.e. supports_device_write returns false.

Parameters

gpu_data	Pointer to the buffer to be written into the sink object
size	Number of bytes to write
stream	CUDA stream to use

Definition at line 163 of file data_sink.hpp.

device_write_async()

virtual std::future<void> cudf::io::data_sink::device_write_async ( void const *  gpu_data, size_t  size, rmm::cuda_stream_view  stream  )

inlinevirtual

Asynchronously append the buffer content to the sink from a gpu address.

For optimal performance, should only be called when is_device_write_preferred returns true. Data sink implementations that don't support direct device writes don't need to override this function.

gpu_data must not be freed until this call is synchronized.

auto result = device_write_async(gpu_data, size, stream);
result.wait(); // OR result.get()

Exceptions

cudf::logic_error	the object does not support direct device writes, i.e. supports_device_write returns false.
cudf::logic_error

Parameters

gpu_data	Pointer to the buffer to be written into the sink object
size	Number of bytes to write
stream	CUDA stream to use

Returns

a future that can be used to synchronize the call

Definition at line 190 of file data_sink.hpp.

host_write()

virtual void cudf::io::data_sink::host_write ( void const *  data, size_t  size  )

pure virtual

Append the buffer content to the sink.

Parameters

[in]	data	Pointer to the buffer to be written into the sink object
[in]	size	Number of bytes to write

is_device_write_preferred()

virtual bool cudf::io::data_sink::is_device_write_preferred ( size_t  size ) const

inlinevirtual

Estimates whether a direct device write would be more optimal for the given size.

Parameters

size	Number of bytes to write

Returns

whether the device write is expected to be more performant for the given size

Definition at line 144 of file data_sink.hpp.

supports_device_write()

virtual bool cudf::io::data_sink::supports_device_write ( ) const

inlinevirtual

Whether or not this sink supports writing from gpu memory addresses.

Internal to some of the file format writers, we have code that does things like

tmp_buffer = alloc_temp_buffer(); cudaMemcpy(tmp_buffer, device_buffer, size); sink->write(tmp_buffer, size);

In the case where the sink type is itself a memory buffered write, this ends up being effectively a second memcpy. So a useful optimization for a "smart" custom data_sink is to do it's own internal management of the movement of data between cpu and gpu; turning the internals of the writer into simply

sink->device_write(device_buffer, size)

If this function returns true, the data_sink will receive calls to device_write() instead of write() when possible. However, it is still possible to receive write() calls as well.

Returns

If this writer supports device_write() calls

Definition at line 136 of file data_sink.hpp.

---

The documentation for this class was generated from the following file:

• data_sink.hpp