Skip to main content

It’s a Wrap for SNIA and the Solid State Storage Initiative at Flash Memory Summit 2019

Marty Foltyn Leave a comment

A Best of Show award, over 12 hours of content, three days of demos, and a new program drawing attention to persistent memory programming completed – Flash Memory Summit 2019 is officially a success!

SNIA volunteers were again recognized for their hard work developing standards for datacenters and storage professionals with a “Most Innovative Flash Memory Technology” FMS Best of Show award. This year, it was SNIA’s Object Drive Technical Work Group who received kudos for the SNIA Technical Position Key Value Storage API Specification.  Jay Kramer, head of the FMS awards program, presented the award to Bill Martin, Chair of the Object Drive TWG, commenting “Key value store technology can enable NVM storage devices to map and store data more efficiently and with enhanced performance, which is of paramount significance to facilitate computational storage.  Flash Memory Summit is proud to recognize the SNIA Object Drive Technical Work Group (TWG) for creating the SNIA Technical Position Key Value Storage API Specification Version 1.0 defining an application programming interface (API) for key value storage devices and making this available to the public for download.

Read More

Your Questions Answered – Now You Can Be a Part of the Real World Workload Revolution!

Marty Foltyn Leave a comment

The SNIA Solid State Storage Initiative would like to thank everyone who attended our webcast: How To Be Part of the Real World Workload Revolution.  If you haven’t seen it yet, you can view the on demand version here.  You can find the slides here.

Eden Kim and Jim Fister led a discussion on the testmyworkload (TMW) tool and data repository, discussing how a collection of real-world workload data captures can revolutionize design and configuration of hardware, software and systems for the industry.   A new SNIA white paper available in both English and Chinese authored by Eden Kim, with an introduction by Tom Coughlin of Coughlin Associates and Jim Handy of Objective Analysis, discusses how we can all benefit by sharing traces of our digital workloads through the SNIA SSSI Real-World Workload Capture program.

Read More

SNIA at Flash Memory Summit 2019 – Your Guide Here!

Marty Foltyn Leave a comment

SNIA technical work and education advances will play a prominent role in the program at the 2019 Flash Memory Summit, August 5-8, 2019, in Santa Clara, CA.  Over 40 speakers will present on key standards activities and education initiatives, including the first ever FMS Persistent Memory Hackathon hosted by SNIA.  Check out your favorite technology (or all), and learn what SNIA is doing in these sessions:

Read More

Register for the PIRL Conference Today

Marty Foltyn Leave a comment

Registration is now open for the upcoming Persistent Programming in Real Life (PIRL) Conference – July 22-23, 2019 on the campus of the University of California San Diego (UCSD).

The 2019 PIRL event features a collaboration between UCSD Computer Science and Engineering, the Non-Volatile Systems Laboratory, and the SNIA to bring industry leaders in programming and developing persistent memory applications together for a two-day discussion on their experiences.

Read More

Calling All Real-World Workloads

Marty Foltyn Leave a comment

Video streaming is an easy-to-understand workload from the I/O perspective, right?  It’s pretty obvious that it’s a workload heavy on long, streaming reads. The application can be modeled with a consistent read flow, and the software tests should be easy.  However, an analysis of the real-world workload shows something very different. At the disk level, the reads turn out to be a rapid flow of 4k and 8k block reads from a solid-state-disk.  Further, other processes on the system also add in a small amount of 4k and 8k writes in the midst of the reads. All of this impacts the application –and an SSD — which was likely heavily tested on the basis of long, streaming reads.

Understanding the real-world characteristics of a workload can be a significant advantage in the development of new hardware, new systems, and new applications.   The SNIA Solid State Storage Initiative (SSSI) and SSSI member company Calypso Systems are providing an opportunity to build a repository of workloads for the industry to use for real-world testing, as outlined in a new SSSI white paper How to Be a Part of the Real-World Workload Revolution. This paper is also available in Chinese at the SSSI Knowledge Center White Papers page.

By going to the TestMyWorkload site, anyone can participate by providing a trace capture of an I/O workload that can be used by others to develop better products. The capture itself traces the block transfers, but does not capture actual data.  Any workload replay would use representative blocks, so there are no concerns about data security or integrity from these captures.

The repository can be used by any participant to test hardware and software, and can help system vendors and users optimize configurations for the best performance based on real-world data.  By participating in this effort, organizations and individuals can provide insight and gain from the knowledge of all the contributors.

Follow these three steps to be a part of the revolution today!

1.  Read the white paper.

2.  Download the free capture tools at TestMyWorkload.com.

3. Mark your calendar and register HERE to learn more in the free SNIA webcast How to Be a Part of the Real-World Workload Revolution on July 9 at 11:00 am Pacific/2:00 pm Eastern.

New Conference Seeking PIRLs of Wisdom

Marty Foltyn Leave a comment

UCSD Computer Science and Engineering, the Non-Volatile Systems Laboratory, and the Storage Networking Industry Association (SNIA) are inviting submissions of proposals for presentation at the first annual Persistent Programming in Real Life (PIRL) conference.  PIRL brings together software development leaders interested in learning about programming methodologies for persistent memories and sharing their experiences with others. This is a meeting for developer project leads on the front lines of persistent programming; not sales, marketing, or non-technical management.

PIRL is small, with attendance limited to under 100 people, including speakers.  It will discuss what real developers have done, and want to do, with persistent memory.  It will involve what worked, what didn’t, what was easy and hard, what was surprising, and what others can learn from the experience.  Presenters are encouraged, and even expected, to show and write code live in the presentation in a comfortable and dynamic peer environment.

Possibilities for presentations include, but are not limited to:

•  Experiences on a particular project

•  Live code development showing new concepts

•  Code challenges

•  New tools for programming

All attendees will be provided access to a development environment to respond to code challenges, or to show their own work in small forums.  This is intended to be a competition-free atmosphere for peers to network with each other to advance the use of persistent memory in the industry and academia.  By combining many of the industry leaders with the academic lights driving practical applications of new technology, peers at PIRL will encourage forward progress for adoption of persistent memory in the marketplace.

Keynote speakers include key personnel from Dreamworks, VMWare, Oracle, Eideticom, and Intel.

PIRL will be hosted by the Non-Volatile Systems Laboratory at the University of California, San Diego.  It will be held at Scripps Forum on July 22nd to 23rd, 2019, with optional events starting July 21st. Pre-registration will be $400.

We’re excited to present this new conference, and we’re excited for you to participate.  Submit your presentation or code challenge idea today. Submissions are due by Monday, June 10th.

 

Trends in Media and Entertainment Storage – Your Questions Answered from Our Webcast

Marty Foltyn Leave a comment

Thanks to all who attended or listened on-demand to our recent SNIA Solid State Storage Initiative (SSSI) webcast on Trends in Worldwide Media and Entertainment Storage. Motti Beck of Mellanox Technologies and Tom Coughlin, SSSI Education Chair and analyst with Coughlin Associates, got rave reviews for their analysis of this important market.  Feedback comments included “Good overview with enough details for me to learn something”; “Really appreciate the insight into the ME businesses”; and “Just in time for the upcoming NAB Show!”.  We appreciate your interest and enthusiasm!

Important to every SNIA webcast are the Questions – and we got quite a few on this one.  Thanks in advance to Tom Coughlin, who provided the answers below.  Send any more questions to us at asksssi@snia.org with the subject- M&E Webcast Questions.  Happy reading, and we hope to see you at one of our upcoming webcasts or events.

 

Q.  What is the best form to store(age) the format video, NAS or SAN?

A.  Well, it depends.  A SAN can directly access the data blocks that make up the video file, these can be quickly transported to the workstation.  There they are reassembled into the video file.  A properly configured SAN can provide faster access to data, particularly if many users are accessing the same data in the storage system.  SANs can be appropriate for a larger production facility.  A NAS may provide somewhat slower access, but provides individual access to individual files.  NAS storage can be an appropriate shared storage for smaller production facilities where there are fewer users or the users don’t access the same files at the same time.

 

Q.  Are there an M&E-specific performance benchmarks or other qualification tools recommended for storage subsystem selection?

A.  That is an interesting question.  I know about several general storage performance benchmarks, such as SPC (https://spcresults.org/benchmarks).  There are storage performance tests offered by some M&E industry suppliers, such as one from AJA System Test (https://www.aja.com/products/aja-system-test).  This is probably an area that could use some additional development.

 

Q.  Revenue share by type and use case – is this on the decline or rise?  What are the YoY trends?

A.  If I understand this right, you are asking questions about revenue growth for different media and entertainment use cases or different parts of the workflow on an annual basis.  That information is in the 2018 Digital Storage for Media and Entertainment Report (https://tomcoughlin.com/tech-papers/)

 

Q.  A question for Tom Coughlin.  You  said 66% will use private or public cloud for archiving in 2018.  Do you have the breakdown between the the two?

A.  It is a combination but given the concerns of the industry, I suspect this is mostly private cloud in 2018.

 

Q.  When Tom says “post production” storage, is that primary, secondary/nearline, or both?

A.  If I understand the question right, this is all storage used in post-production, which can use a primary and secondary storage tier, particularly in a larger facility which is economizing on its storage costs.

 

Q.  With regard to HDD storage, does the interface trend continue to be SATA/SAS?  Does the back end workload look to benefit from SMR or dual-actuator technology?

A.  For the time being HDDs will be SATA and SAS.  There are now some HDD storage systems with NVMe on the back end and it will be interesting to see how this develops.  I am sure that M&E users will benefit from SMR and dual-actuator HDDs.  SMR will be good for active archiving in particular and dual actuator will allow faster access to HDD data, a benefit for video projects.

 

Q.  Unless I missed it, you made no mention of software-defined storage as a viable method for storing the growing amount of data in M&E.  Was that taken into consideration when you did your survey?

A.  Software defined storage can be an important element in media and entertainment storage and is finding increasing use in this and other applications.

 

Q.  Is the cloud archive, primary copy or secondary (insurance policy with limited to no access)?

A.  It depends upon the organization, although I think for many studios and larger organizations, they may keep content on tape and even off-line tape as well.  Cloud archives do allow access to data, the usual issue is the cost of egressing that content.

Your Questions Answered – Applications Take Advantage of Persistent Memory Webcast

Marty Foltyn Leave a comment

We hope you had time to check out our recent webcast on Applications Take Advantage of Persistent Memory.  Raghu Kulkarni of Viking Technology, a member of the SNIA Solid State Storage Initiative, did a great job laying the foundation for an understanding of Persistent Memory today, just in time for the SNIA Persistent Memory Summit.

You can catch up on videos of Summit talks, along with the slides presented, here.

During the webcast, we had many interesting questions.  Now, as promised, Raghu provides the answers.  Happy reading, and we hope to see you at one of our upcoming webcasts or events.

Q.  Does NVDIMM-N encryption lower the performance levels that you presented?

A.  It typically depends on the implementation and differs from each vendor. Generally speaking, Save and Restore operations will increase by a small factor – less than 10%.  Products from some vendors, like Viking, will not see a performance degradation as it is offset by a faster transfer rate

Q.  What are the read/write bandwidth capabilities of NVDIMM-N? How does that compare to Intel’s Persistent Memory?

A.  For Byte-addressable mode, NVDIMM-N in theory has the same high performance as DRAM, around 100ns. With the latest Linux drivers in DAX mode, NVDIMM-N are still expected to be better than Intel’s Persistent Memory.

Q.  On the use cases, what are the use cases when Persistent Memory is attached to an accelerator chip compared to a Processor attached setup?

A.  Mainly to accelerate the performance by storing the metadata or even data in Persistent Memory, so that the request can be acknowledged immediately without having to wait for commits to SSD/HDD. It also saves the rebuild time, which is a common practice for volatile memory.

Q.  How does BIOS/MRC work when a Persistent Memory is attached to an accelerator (ASIC/FPGA/GPU) over PCIe, when trying to extended/increase the memory for the processor?

A.  System BIOS will not detect the Persistent Memory sitting on PCIe; it only discovers Persistent Memory installed in DIMM slots. FPGA/ASIC, etc. have to build their own bottom up code to detect and present the Persistent Memory on PCIe depending on the use case.

Q.  Do we need application changes to take advantage of Persistent Memory-aware file storage? how does it compare against the DAX mode?

A.  To take advantage of the low latency/high performance nature of Persistent Memory, it would be beneficial to modify the applications. However, one can still leverage the existing IO stack if modifying the application is not an option. Check out pmem.io for pre-built libraries that can be directly integrated into applications.

Q.  Should the Persistent Memory usage be compared against the Storage or Memory. Which is a more relevant use case for Persistent Memory?

A.  Typically, a media that is Byte-addressable is called Persistent Memory (PM); however, you can also access it in Block mode. Again, depending on the application needs, use case, and other system level factors it can be used in either modes.  However, you will find best performance when accessing in Byte-addressable/Load-Store mode.

 

 

Innovating File System Architectures with NVMe

Marty Foltyn Leave a comment

It’s exciting to see the recent formation of the Solid State Drive Special Interest Group (SIG) here in the SNIA Solid State Storage Initiative.  After all, everyone appreciates the ability to totally geek out about the latest drive technology and software for file systems.  Right? Hey, where’s everyone going? We have vacation pictures with the dog we stored that we want to show…

Solid state storage has long found its place with those seeking greater performance in systems, especially where smaller or more random block/file transfers are prevalent.  Single-system opportunity with NVMe drives is broad, and pretty much unquestioned by those building systems for the modern IT environments. Cloud, likewise, has found use of the technology where single-node performance makes a broader deployment relevant.

There have been many efforts to build the case for solid state in networked storage.  Where storage and computation combine — for instance in a large map/reduce application — there’s been significant advantage, especially in the area of sustained data reads.  This has usually comes at a scalar cost, where additional systems are needed for capacity. Nonetheless, finding cases where non-volatile memory enhances infrastructure deployment for storage or analytics.  Yes, analytics is infrastructure these days, deal with it.

Seemingly independent of the hardware trends, the development of new file systems has provided significant innovation.  Notably, heavily parallel file systems have the ability to serve a variety of network users in specialized applications or appliances.  Much of the work has focused on development of the software or base technology rather than delivering a broader view of either performance or applicability.  Therefore, a paper such as this one on building a Lustre file system using NVMe drives is a welcome addition to the case for both solid state storage and revolutionary file systems that move from specific applications to more general availability.

The paper shows how to build a small (half-rack) cluster of storage to support the Lustre file system, and it also adds the Dell VFlex OS implemented as a software defined storage solution.  This has the potential to take an HPC-focused product like Lustre and drive a broader market availability for a high-performance solution. The combination of read/write performance, easy adoption to the broad enterprise, and relatively small footprint shows new promise for innovation.

The opportunity for widespread delivery of solid state storage using NVMe and software innovation in the storage space is ready to move the datacenter to new and more ambitious levels.  The SNIA 2019 Storage Developer Conference  is currently open for submissions from storage professionals willing to share knowledge and experience.  Innovative solutions such as this one are always welcome for consideration.

Hacking with the U

Marty Foltyn Leave a comment

It’s now less than three weeks for the next SNIA Persistent Memory Hackathon and Workshop.  Our next workshop will be held in conjunction with the 10th Annual Non-Volatile Memory Workshop (http://nvmw.ucsd.edu/) at the University of California, San Diego on Sunday, March 10th from 2:00pm to 5:30pm.

The Hackathon at NVMW19 provides software developers with an understanding of the different tiers and modes of persistent memory, and gives an overview of the standard software libraries that are available to access persistent memory.  Attendees will have access to system configured with persistent memory, software libraries, and sample source code. A variety of mentors will be available to provide tutorials and guide participants in the development of code. Learn more here.

In the last workshop, the feedback from the attendees pointed to a desire to work longer on code after the tutorial ended.  We will ensure that all the Hackathon attendees will have access to their environment through the length of the conference. So any participant in the Sunday session will be able to continue work until the conference completion on Tuesday afternoon.  While there won’t be an opportunity for formal follow-up, we’re planning an informal meet-up the final day of the conference. Stay tuned for details.

For those not familiar with NVMW, the program is replete with the latest in non-volatile memory research, which enables attendees to understand the practical advances in software development for persistence.  The workshop facilitates the exchange of ideas and advances collaboration and knowledge of the use of persistent memory. Registration for the conference is affordable, and grants are available for university student attendees.

For those not able to get to San Diego in March, enjoy the weather that obviously won’t be anywhere near as nice where you live.  Oh, sorry. For those not able to get to San Diego in March, SNIA is working on the next opportunities for a formal hackathon. But we can’t do it alone.  If you have a group of programmers interested in learning persistent memory development, SNIA would consider coming to you with a Host a Hackathon. We can provide, or even train, mentors to get you started, and show you how to build your own cloud-based development environment.  You’ll get an introduction to coding, and you’ll be left with some great examples to build your own applications. Contact SNIA at PMhackathon@snia.org for more details and visit our PM Programming Hackathon webpage for the latest updates.