You can import CSV files directly into an SQLite database. Then you are free to do whatever sql searches or manipulations you want. Python and golang both have great SQLite libraries from personal experience, but I’d be surprised if there is any language that doesn’t have a decent one.
If you are running Linux most distros have an SQLite gui browser in the repos that is pretty powerful.
I’d be surprised if there is any language that doesn’t have a decent one.
Yeah, SQLite provides a library implemented in C. Because C doesn’t require a runtime, it’s possible for other languages to call into this C library. All you need is a relatively thin wrapper library, which provides an API that feels good to use in the respective language.
I love self-hosted tools, but you can do a lot on a spreadsheet.
Btw, if the files are not too large, you can query them using SQL without even hosting a database just by using Pandas. This avoids the problem of updating entries and handling migrations in case the CSVs change over time.
Depends on the size of data, use case like will you be doing any constant updates to it or just reading, you mentioned you have several files so do you need joins if so what will be an approx max number of joins you’ll be doing on per query basis, I guess you said CSV so I’m assuming it is structured data and not semi structured or unstructured.
Few more questions, do you need a fast indexing but are not planning on doing any complex operations, areyoiu going to do a lot of OLTP operations and you need ACID. Or are you going OLAP route. are you planning on distributed database if so then which 2 do you want from CAP, do you want batch processing or stream processing,
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhost@lemmy.ml
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules
No harassment
crossposts from c/Open Source & c/docker & related may be allowed, depending on context
Video Promoting is allowed if is within the topic.
You can import CSV files directly into an SQLite database. Then you are free to do whatever sql searches or manipulations you want. Python and golang both have great SQLite libraries from personal experience, but I’d be surprised if there is any language that doesn’t have a decent one.
If you are running Linux most distros have an SQLite gui browser in the repos that is pretty powerful.
Yeah, SQLite provides a library implemented in C. Because C doesn’t require a runtime, it’s possible for other languages to call into this C library. All you need is a relatively thin wrapper library, which provides an API that feels good to use in the respective language.
Import it into access(Satan whispers quietly into you ear)
Excel / OnlyOffice?
I love self-hosted tools, but you can do a lot on a spreadsheet.
Btw, if the files are not too large, you can query them using SQL without even hosting a database just by using Pandas. This avoids the problem of updating entries and handling migrations in case the CSVs change over time.
Files won’t change and are hundreds of GBs
ok, database it is then
Are they roughly 55GB compressed?
Spill the beans!
👀
🕵️
Depends on the size of data, use case like will you be doing any constant updates to it or just reading, you mentioned you have several files so do you need joins if so what will be an approx max number of joins you’ll be doing on per query basis, I guess you said CSV so I’m assuming it is structured data and not semi structured or unstructured.
Few more questions, do you need a fast indexing but are not planning on doing any complex operations, areyoiu going to do a lot of OLTP operations and you need ACID. Or are you going OLAP route. are you planning on distributed database if so then which 2 do you want from CAP, do you want batch processing or stream processing,
I’ve few dozen other questions also