Skip to content

Commit

Permalink
Fix build on x86 and bump cargo doc version as packed_simd fails on d…
Browse files Browse the repository at this point in the history
…ocs.rs version
  • Loading branch information
alecmocatta committed Sep 25, 2018
1 parent 935405c commit 283183c
Show file tree
Hide file tree
Showing 5 changed files with 32 additions and 31 deletions.
15 changes: 1 addition & 14 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ authors = ["Alec Mocatta <alec@mocatta.net>"]
categories = ["data-structures","algorithms","science"]
keywords = ["streaming-algorithm","probabilistic","sketch","data-structure","hyperloglog"]
description = """
Performant implementations of various streaming algorithms, including Count–min sketch; Top k; HyperLogLog; Reservoir sampling.
SIMD-accelerated implementations of various streaming algorithms, including Count–min sketch, Top k, HyperLogLog, Reservoir sampling.
"""
repository = "https://github.com/alecmocatta/streaming_algorithms"
homepage = "https://github.com/alecmocatta/streaming_algorithms"
Expand All @@ -26,17 +26,4 @@ twox-hash = "1.1"
serde_derive = "1.0"
serde = "1.0"
rand = "0.5"
bytecount = "0.3"
# faster = { git = "https://github.com/alecmocatta/faster" } #"0.5"
# faster = { path = "./faster" } #"0.5"
packed_simd = { version = "0.3", features = ["into_bits"] }

[features]
avx-accel = ["bytecount/avx-accel"]
simd-accel = ["bytecount/simd-accel"]

# [replace]
# "vektor:0.2.0" = { git = "https://github.com/alecmocatta/vektor" }

[profile.release]
debug = 2
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

[Docs](https://docs.rs/streaming_algorithms/0.1.0)

Performant implementations of various [streaming algorithms](https://en.wikipedia.org/wiki/Streaming_algorithm).
SIMD-accelerated implementations of various [streaming algorithms](https://en.wikipedia.org/wiki/Streaming_algorithm).

This library is a work in progress. PRs are very welcome! Currently implemented algorithms include:

Expand All @@ -17,7 +17,13 @@ This library is a work in progress. PRs are very welcome! Currently implemented
* HyperLogLog
* Reservoir sampling

A goal of this library is to enable composition of these algorithms; for example Top k + HyperLogLog to enable roughly `SELECT key FROM table GROUP BY key ORDER BY COUNT(DISTINCT value) DESC LIMIT k`.
A goal of this library is to enable composition of these algorithms; for example Top k + HyperLogLog to enable an approximate version of something akin to `SELECT key FROM table GROUP BY key ORDER BY COUNT(DISTINCT value) DESC LIMIT k`.

Run your application with `RUSTFLAGS="-C target-cpu=native"` to benefit from the SIMD-acceleration like so:

```bash
RUSTFLAGS="-C target-cpu=native" cargo run --release
```

See [this gist](https://gist.github.com/debasishg/8172796) for a good list of further algorithms to be implemented. Other resources are [Probabilistic data structures – Wikipedia](https://en.wikipedia.org/wiki/Category:Probabilistic_data_structures), [DataSketches – A similar Java library originating at Yahoo](https://datasketches.github.io/), and [Algebird – A similar Java library originating at Twitter](https://github.com/twitter/algebird).

Expand Down
22 changes: 13 additions & 9 deletions src/distinct.rs
Original file line number Diff line number Diff line change
Expand Up @@ -392,29 +392,33 @@ mod simd_types {
use self::simd_types::*;

struct Sad<X>(PhantomData<fn(X)>);
#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
mod x86 {
#[cfg(target_arch = "x86")]
pub use std::arch::x86::*;
#[cfg(target_arch = "x86_64")]
pub use std::arch::x86_64::*;
}
// TODO
// #[cfg(target_feature = "avx512bw")]
// impl Sad<packed_simd::u8x64> {
// #[inline]
// #[target_feature(enable = "avx512bw")]
// unsafe fn sad(a: packed_simd::u8x64, b: packed_simd::u8x64) -> packed_simd::u64x8 {
// use std::{arch::x86_64::_mm512_sad_epu8, mem::transmute};
// packed_simd::Simd(transmute(_mm512_sad_epu8(transmute(a.0), transmute(b.0))))
// use std::mem::transmute;
// packed_simd::Simd(transmute(x86::_mm512_sad_epu8(transmute(a.0), transmute(b.0))))
// }
// }
mod x86 {
#[cfg(target_arch = "x86")]
pub use std::arch::x86::*;
#[cfg(target_arch = "x86_64")]
pub use std::arch::x86_64::*;
}
#[cfg(target_feature = "avx2")]
impl Sad<packed_simd::u8x32> {
#[inline]
#[target_feature(enable = "avx2")]
unsafe fn sad(a: packed_simd::u8x32, b: packed_simd::u8x32) -> packed_simd::u64x4 {
use std::mem::transmute;
packed_simd::Simd(transmute(x86::_mm256_sad_epu8(transmute(a.0), transmute(b.0))))
packed_simd::Simd(transmute(x86::_mm256_sad_epu8(
transmute(a.0),
transmute(b.0),
)))
}
}
#[cfg(target_feature = "sse2")]
Expand Down
14 changes: 9 additions & 5 deletions src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//! Performant implementations of various [streaming algorithms](https://en.wikipedia.org/wiki/Streaming_algorithm).
//! SIMD-accelerated implementations of various [streaming algorithms](https://en.wikipedia.org/wiki/Streaming_algorithm).
//!
//! **[Crates.io](https://crates.io/crates/streaming_algorithms) │ [Repo](https://github.com/alecmocatta/streaming_algorithms)**
//!
Expand All @@ -9,7 +9,13 @@
//! * HyperLogLog
//! * Reservoir sampling
//!
//! A goal of this library is to enable composition of these algorithms; for example Top k + HyperLogLog to enable roughly `SELECT key FROM table GROUP BY key ORDER BY COUNT(DISTINCT value) DESC LIMIT k`.
//! A goal of this library is to enable composition of these algorithms; for example Top k + HyperLogLog to enable an approximate version of something akin to `SELECT key FROM table GROUP BY key ORDER BY COUNT(DISTINCT value) DESC LIMIT k`.
//!
//! Run your application with `RUSTFLAGS="-C target-cpu=native"` to benefit from the SIMD-acceleration like so:
//!
//! ```bash
//! RUSTFLAGS="-C target-cpu=native" cargo run --release
//! ```
//!
//! See [this gist](https://gist.github.com/debasishg/8172796) for a good list of further algorithms to be implemented. Other resources are [Probabilistic data structures – Wikipedia](https://en.wikipedia.org/wiki/Category:Probabilistic_data_structures), [DataSketches – A similar Java library originating at Yahoo](https://datasketches.github.io/), and [Algebird – A similar Java library originating at Twitter](https://github.com/twitter/algebird).
//!
Expand Down Expand Up @@ -54,11 +60,9 @@
extern crate twox_hash;
#[macro_use]
extern crate serde_derive;
// extern crate bytecount;
extern crate packed_simd;
extern crate rand;
extern crate serde;
// extern crate faster;
extern crate packed_simd;

mod count_min;
mod distinct;
Expand Down
2 changes: 1 addition & 1 deletion src/top.rs
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ mod test {
// println!("{:#?}", x);
}

// #[ignore] // takes too long on CI
#[ignore] // takes too long on CI
#[test]
fn many() {
let start = time::Instant::now();
Expand Down

0 comments on commit 283183c

Please sign in to comment.