NRW.social

0 Beiträge0 Beteiligte0 Beiträge heute

Fabian<a href="https://mstdn.science/@ChristosArgyrop" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@ChristosArgyrop</a> While this has truth, there is still <a href="https://mastodon.social/tags/DuckDB" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DuckDB</a> and <a href="https://mastodon.social/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a> that can help tons with large datasets before we need actual SQL! Don't know if that's counting as giggles though 😅

Data QuineMy browser WASM’t prepared for this. Using DuckDB, Apache Arrow and Web Workers in real life | by Motif Analytics | Feb, 2025 | Medium <a href="https://motifanalytics.medium.com/my-browser-wasmt-prepared-for-this-using-duckdb-apache-arrow-and-web-workers-in-real-life-e3dd4695623d" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://motifanalytics.medium.com/my-browser-wasmt-prepared-for-this-using-duckdb-apache-arrow-and-web-workers-in-real-life-e3dd4695623d</a><a href="https://datasci.social/tags/DuckDB" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DuckDB</a> <a href="https://datasci.social/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a> <a href="https://datasci.social/tags/DataAnalysis" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataAnalysis</a>

André Ourednik<a href="https://mastodon.social/tags/apachearrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#apachearrow</a> and <a href="https://mastodon.social/tags/gdal" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#gdal</a> both rely on aws-c-cmmon and related packagesHaving no trust in anything <a href="https://mastodon.social/tags/Amazon" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Amazon</a> related (though there are certainly good people at AWS Labs, too), a question:Isn't there a way to make arrow and gdal depend on some other packages?<a href="https://fosstodon.org/@jorisvandenbossche" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@jorisvandenbossche</a> <a href="https://mastodon.social/@gdal" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@gdal</a>

amoeba🏹 We’re excited to announce the release of {arrow} 17.0.0.1. Binary packages are now available for all platforms from both CRAN and R-Universe (<a href="https://apache.r-universe.dev/arrow" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://apache.r-universe.dev/arrow</a>). This release includes some nice quality of life improvements for folks writing dplyr pipelines with arrow. Have a look below to see what’s changed or see the full changelog <a href="https://arrow.apache.org/docs/r/news/index.html" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://arrow.apache.org/docs/r/news/index.html</a> for all the info. <a href="https://toot.cafe/tags/rstats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rstats</a> <a href="https://toot.cafe/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a>

Nic CraneWe're delighted to announce that "Scaling Up with R and Arrow", by Nic Crane, Jonathan Keane and Neal Richardson is now available online at <a href="http://www.arrowrbook.com" rel="nofollow noopener noreferrer" translate="no" target="_blank">http://www.arrowrbook.com</a>. In the book, we cover a lot of the practical details and theory behind working with Arrow in R. The paper version will be available soon! <a href="https://mastodon.social/tags/rstats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rstats</a> <a href="https://mastodon.social/tags/apachearrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#apachearrow</a>

Alejandro BaezMessing about with <a href="https://fosstodon.org/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a> recently. And ended up discovering <a href="https://fosstodon.org/tags/glaredb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#glaredb</a>. A <a href="https://fosstodon.org/tags/rust" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rust</a> embedded db, in the style of <a href="https://fosstodon.org/tags/duckdb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#duckdb</a>. 😎 Though I can't say much on it yet, as I haven't yet fully grasped you wouldn't use duckdb instead. 😅Still. Very much liking competition in this new age <a href="https://fosstodon.org/tags/OLAP" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OLAP</a> space. 😄<a href="https://glaredb.com/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://glaredb.com/</a>

rickspencer3<a href="https://gladtech.social/@cuchaz" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@cuchaz</a> have you taken a look at <a href="https://social.lol/tags/apachearrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#apachearrow</a> and parquet?

Stas KolenikovI am working in a project that has bits of <a href="https://mastodon.online/tags/rstats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rstats</a>, bits of <a href="https://mastodon.online/tags/python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#python</a> and bits of <a href="https://mastodon.online/tags/cpp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#cpp</a> and they are all supposed to be united by <a href="https://mastodon.online/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a>. How do I access externally created Arrow objects in R? Everything that is in <a href="https://r4ds.hadley.nz/arrow.html" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://r4ds.hadley.nz/arrow.html</a>, <a href="https://arrow-user2022.netlify.app/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://arrow-user2022.netlify.app/</a> (with huge thanks to <a href="https://hachyderm.io/@djnavarro" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@djnavarro</a>), <a href="https://arrow.apache.org/cookbook/r" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://arrow.apache.org/cookbook/r</a> (thanks to <a href="https://fosstodon.org/@nic_crane" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@nic_crane</a>) talks about creating large Arrow-ish objects in R by reading external data in csv or parquet rather than connecting to the existing Arrow sources.

Jacob ScottHey <a href="https://fosstodon.org/tags/RStats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RStats</a>, keen to get takes on this.I really like data-management solutions like <a href="https://fosstodon.org/tags/duckdb" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#duckdb</a>, <a href="https://fosstodon.org/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a>, <a href="https://fosstodon.org/tags/sqlite" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sqlite</a> etc which help you manage large datasets whilst keeping your analysis local.My question is, how do you approach version control with these tools? How do you make your work reproducible? Not talking about your code (just use Git), but the data it operates on. I can create a duckdb database, but it's not obvious how I should share this with others.<a href="https://fosstodon.org/tags/DataScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataScience</a> <a href="https://fosstodon.org/tags/Database" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Database</a> <a href="https://fosstodon.org/tags/sql" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#sql</a>

FOSSlifeInaugural recipients of newly-established FOSS Contributor Fund announced by Bloomberg <a href="https://www.fosslife.org/bloomberg-announces-first-recipients-new-foss-fund" rel="nofollow noopener noreferrer" target="_blank">https://www.fosslife.org/bloomberg-announces-first-recipients-new-foss-fund</a> <a href="https://fosstodon.org/tags/FOSS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FOSS</a> <a href="https://fosstodon.org/tags/funding" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#funding</a> <a href="https://fosstodon.org/tags/Bloomberg" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Bloomberg</a> <a href="https://fosstodon.org/tags/software" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#software</a> <a href="https://fosstodon.org/tags/OpenSource" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#OpenSource</a> <a href="https://fosstodon.org/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a> <a href="https://fosstodon.org/tags/Curl" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Curl</a> <a href="https://fosstodon.org/tags/Celery" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Celery</a>

Sharon Machlis<a href="https://social.coop/@eamon" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@eamon</a> Although you can use :rstats: for data that won't fit in memory too 😀 <a href="https://fosstodon.org/@thomas_mock" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@thomas_mock</a> 's lightning talk at last year's Arrow conference Video <a href="https://www.youtube.com/watch?v=LvTX1ZAZy6M" rel="nofollow noopener noreferrer" target="_blank">https://www.youtube.com/watch?v=LvTX1ZAZy6M</a> Slides <a href="https://jthomasmock.github.io/arrow-dplyr/#/" rel="nofollow noopener noreferrer" target="_blank">https://jthomasmock.github.io/arrow-dplyr/#/</a> <a href="https://fosstodon.org/tags/rstats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rstats</a> <a href="https://fosstodon.org/tags/DuckDB" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DuckDB</a> <a href="https://fosstodon.org/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a>

Daniel HockingI was just recommending <a href="https://fosstodon.org/@djnavarro" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@djnavarro</a> posts about <a href="https://bayes.club/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a> <a href="https://bayes.club/tags/Parquet" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Parquet</a> and <a href="https://bayes.club/tags/DataScience" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DataScience</a> in <a href="https://bayes.club/tags/RStats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RStats</a> to someone and decided I'd share here too. She writes fantastic intros and explanations in entertaining posts, tutorials, and courses.<a href="https://blog.djnavarro.net/posts/2021-11-19_starting-apache-arrow-in-r/" rel="nofollow noopener noreferrer" target="_blank">https://blog.djnavarro.net/posts/2021-11-19_starting-apache-arrow-in-r/</a><a href="https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-datasets/" rel="nofollow noopener noreferrer" target="_blank">https://blog.djnavarro.net/posts/2022-11-30_unpacking-arrow-datasets/</a><a href="https://arrow-user2022.netlify.app/" rel="nofollow noopener noreferrer" target="_blank">https://arrow-user2022.netlify.app/</a>

Gunnar Morling🗣️ "The central idea behind Flight is deceptively simple: it provides a standard protocol for transferring Arrow data over a network"Great post by <a href="https://fosstodon.org/@djnavarro" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@djnavarro</a>; <a href="https://mastodon.online/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a> Flight (and Flight SQL) is super-interesting, definitely keep an eye on it in '23.<a href="https://blog.djnavarro.net/posts/2022-10-18_arrow-flight/" rel="nofollow noopener noreferrer" target="_blank">https://blog.djnavarro.net/posts/2022-10-18_arrow-flight/</a>

Danielle NavarroFor reasons unknown she is blogging again. I am so sorry, but should you happen to be curious about how Dataset objects work in the <a href="https://fosstodon.org/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a> <a href="https://fosstodon.org/tags/RStats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RStats</a> package, and enjoy me being mildly irritable about... things, this post may be of some interest? :blobcatheart: <a href="https://blog.djnavarro.net/unpacking-arrow-datasets" rel="nofollow noopener noreferrer" target="_blank">https://blog.djnavarro.net/unpacking-arrow-datasets</a>

Kae SuarezHello! I'm Kae, and this is my Hachyderm. I want to be clear about what's going to happen here. I am going to be making a lot of posts that are me floundering with tech. That's the point. As a person in DevRel, I think it's important to be honest about floundering now and then, and let others, especially engineers, see some pain points. Plus, it can be entertaining, and gives me tons of draft material for my blog. I hope I can show everyone some cool things!<a href="https://hachyderm.io/tags/python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#python</a> <a href="https://hachyderm.io/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a>

Danielle NavarroMy favourite trick for working with huge data sets in R. If your dataset is larger than memory and the query result is also larger than memory, you can still use dplyr/arrow pipelines. Example:library(arrow) library(dplyr)nyc_taxi <- open_dataset("nyc-taxi/") nyc_taxi |> filter(payment_type == "Credit card") |> group_by(year, month) |> write_dataset("nyc-taxi-credit")Input is 1.7 billion rows (70GB), output is 500 million (15GB). Takes 3-4 mins on my laptop 🙂 <a href="https://fosstodon.org/tags/rstats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rstats</a> <a href="https://fosstodon.org/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a>

Danielle NavarroI am diving into the internals trying to understand how the arrow <a href="https://fosstodon.org/tags/rstats" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#rstats</a> package wraps the <a href="https://fosstodon.org/tags/ApacheArrow" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ApacheArrow</a> Datasets API and the bar is playing "Rebel Girl" by Bikini Kill. Honestly living my best life

Frühere Suchanfragen

Suchoptionen

Verwaltet von:

Serverstatistik:

#ApacheArrow