nested task error: Abstract type AbstractTime does not have a definite size

[lawless-m]

df = DataFrame()
df[!, :test] = [DateTime(2000,1,1,1,1,1), missing] # the missing is causing the problem
DataFrameTools.df_write("test.jdf", df)

ERROR: TaskFailedException
Stacktrace:
[1] wait
@ .\task.jl:322 [inlined]
[2] fetch(t::Task)
@ Base .\task.jl:337
[3] _broadcast_getindex_evalf
@ .\broadcast.jl:648 [inlined]
[4] _broadcast_getindex
@ .\broadcast.jl:621 [inlined]
[5] getindex
@ .\broadcast.jl:575 [inlined]
[6] copy
@ .\broadcast.jl:922 [inlined]
[7] materialize
@ .\broadcast.jl:883 [inlined]
[8] save(outdir::String, df::DataFrame; verbose::Bool)
@ JDF C:\Users\ME.julia\packages\JDF\SMfQY\src\savejdf.jl:82
[9] save
@ C:\Users\ME.julia\packages\JDF\SMfQY\src\savejdf.jl:52 [inlined]
[10] #savejdf#64
@ C:\Users\ME.julia\packages\JDF\SMfQY\src\savejdf.jl:145 [inlined]
[11] savejdf
@ C:\Users\ME.julia\packages\JDF\SMfQY\src\savejdf.jl:145 [inlined]
[12] _df_write(file::String, format::Symbol, df::DataFrame; dictencode::Bool)
@ DataFrameTools C:\Users\ME.julia\packages\DataFrameTools\tsE6U\src\io.jl:188
[13] df_write(file::String, df::DataFrame; subformat::Nothing, dictencode::Bool)
@ DataFrameTools C:\Users\ME.julia\packages\DataFrameTools\tsE6U\src\io.jl:140
[14] df_write(file::String, df::DataFrame)
@ DataFrameTools C:\Users\ME.julia\packages\DataFrameTools\tsE6U\src\io.jl:124
[15] top-level scope
@ REPL[49]:1

nested task error: Abstract type AbstractTime does not have a definite size.                                                                               
Stacktrace:                                                                                                                                                
  [1] sizeof                                                                                                                                               
    @ .\essentials.jl:455 [inlined]                                                                                                                        
  [2] compress!(dest::Vector{UInt8}, src::Ptr{Dates.AbstractTime}, src_size::Int64)                                                                        
    @ Blosc C:\Users\ME\.julia\packages\Blosc\Six7M\src\Blosc.jl:58                                                                             
  [3] compress(src::Ptr{Dates.AbstractTime}, src_size::Int64; kws::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})               
    @ Blosc C:\Users\ME\.julia\packages\Blosc\Six7M\src\Blosc.jl:86                                                                             
  [4] compress(src::Ptr{Dates.AbstractTime}, src_size::Int64)                                                                                              
    @ Blosc C:\Users\ME\.julia\packages\Blosc\Six7M\src\Blosc.jl:85                                                                             
  [5] compress(src::Vector{Dates.AbstractTime}; kws::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})                             
    @ Blosc C:\Users\ME\.julia\packages\Blosc\Six7M\src\Blosc.jl:92                                                                             
  [6] compress(src::Vector{Dates.AbstractTime})                                                                                                            
    @ Blosc C:\Users\ME\.julia\packages\Blosc\Six7M\src\Blosc.jl:91                                                                             
  [7] compress_then_write(b::Vector{Dates.AbstractTime}, io::BufferedStreams.BufferedOutputStream{IOStream})                                               
    @ JDF C:\Users\ME\.julia\packages\JDF\SMfQY\src\compress_then_write.jl:8                                                                    
  [8] compress_then_write(b::Vector{Union{Missing, DateTime}}, io::BufferedStreams.BufferedOutputStream{IOStream})                                         
    @ JDF C:\Users\ME\.julia\packages\JDF\SMfQY\src\type-writer-loader\Missing.jl:13                                                            
  [9] macro expansion                                                                                                                                      
    @ C:\Users\ME\.julia\packages\JDF\SMfQY\src\savejdf.jl:75 [inlined]                                                                         
 [10] (::JDF.var"#57#60"{String, DataFrame, Symbol})()                                                                                                     
    @ JDF .\threadingconstructs.jl:169                                                                                                                     

julia>

[lawless-m]

Luckily for me, it works for Union{Date, Missing} and I don't need the time in my application

[chipkent]

Can you send the output of versioninfo() so that I can look into this?

[lawless-m]

Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i3-3240 CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, ivybridge)

[chipkent]

Reproducer code:

use docker run -it julia:1.6:

using Pkg
Pkg.add("DataFrameTools")
using DataFrameTools
Pkg.add("DataFrames")
using DataFrames
Pkg.add("Dates")
using Dates
df = DataFrame()
df[!, :test] = [DateTime(2000,1,1,1,1,1), missing]
df

DataFrameTools.df_write("test.jdf", df)

sizeof(Date(2000,1,1))
sizeof(DateTime(2000,1,1,1,1,1))
sizeof(DateTime(2000,1,1,1,1,1)::Union{DateTime,Missing})
sizeof(Union{DateTime,Missing})

Pkg.add("JDF")
using JDF
JDF.savejdf("test.jdf", df)

[chipkent]

The root cause of the problem is a bug in JDF:
xiaodaigh/JDF.jl#62

The bug is reported as fixed in 0.4.1. Fix to come shortly.

[chipkent]

After digging in, JDF is already at version 0.4.4. The JDF ticket specifically addresses Date but not DateTime.

The following currently works, as you mentioned above.

col = [Date(1999,1,1),missing,today()]
df = DataFrame(d = col)
JDF.save("test.jdf", df)

[chipkent]

A bug report is filed at:
xiaodaigh/JDF.jl#72

Once there is a fix, the JDF version can be updated.