Remote Persistent Memory: It Takes a Village (or Perhaps a City)

By Paul Grun, Chair, OpenFabrics Alliance and Senior Technologist, Cray, Inc.

Remote Persistent Memory, (RPM), is rapidly emerging as an important new technology. But understanding a new technology, and grasping its significance, requires engagement across a wide range of industry organizations, companies, and individuals. It takes a village, as they say.

Technologies that are capable of bending the arc of server architecture come along only rarely. It’s sometimes hard to see one coming because it can be tough to discern between a shiny new thing, an insignificant evolution in a minor technology, and a serious contender for the Technical Disrupter of the Year award. Remote Persistent Memory is one such technology, the ultimate impact of which is only now coming into view. Two relatively recent technologies serve to illustrate the point: The emergence of dedicated, high performance networks beginning in the early 2000s and more recently the arrival of non-volatile memory technologies, both of which are leaving a significant mark on the evolution of computer systems. But what happens when those two technologies are combined to deliver access to persistent memory over a fabric? It seems likely that such a development will positively impact the well-understood memory hierarchies that are the basis of all computer systems today. And that, in turn, could cause system architects and application programmers to re-think the way that information is accessed, shared, and stored. To help us bring the subject of RPM into sharp focus, there is currently a concerted effort underway to put some clear definition around what is shaping up to be a significant disrupter.

For those who aren’t familiar, Remote Persistent Memory refers to a persistent memory service that is accessed over a fabric or network. It may be a service shared among multiple users, or dedicated to one user or application. It’s distinguished from local Persistent Memory, which refers to a memory device attached locally to the processor via a memory or I/O bus, in that RPM is accessed via a high performance switched fabric. For our purposes, we’ll further refine our discussion to local fabrics, neglecting any discussion of accessing memory over the wide area.

Most important of all, Persistent Memory, including RPM, is definitely distinct from storage, whether that is file, object or block storage. That’s why we label this as a ‘memory’ service – to distinguish it from storage.  The key distinction is that the consumer of the service recognizes and uses it as it would any other level in the memory hierarchy. Even though the service could be implemented using block or file-oriented non-volatile memory devices, the key is in the way that an application accesses and uses the service. This isn’t faster or better storage, it’s a whole different kettle of fish.

So how do we go about discovering the ultimate value of a new technology like RPM? So far, a lively discussion has been taking place across multiple venues and industry events. These aren’t ad hoc discussions nor are they tightly scripted events; they are taking place in a loosely organized fashion designed to encourage lots of participation and keep the ball moving forward. Key discussions on the topic have hopscotched from the SNIA’s Storage Developers Conference, to SNIA/SSSI’s Persistent Memory Summit, to the OpenFabrics Alliance (OFA) Workshop and others. Each of these industry events has given us an opportunity for the community at large to discuss and develop the essential ideas surrounding RPM. The next installment will occur at the upcoming Flash Memory Summit in August where there will be four sessions all devoted to discussing Remote Persistent Memory.

Having frequent industry gatherings is a good thing, naturally, but that by itself doesn’t answer the question of how we go about progressing a discussion of Remote Persistent Memory in an orderly way.  A pretty clear consensus has emerged that RPM represents a new layer in the memory hierarchy and therefore the best way to approach it is to take a top-down perspective. That means starting with an examination of the various ways that an application could leverage this new player in the memory hierarchy. The idea is to identify and explore several key use cases. Of course, the technology is in its early infancy, so we’re relying on the best instincts of the industry at large to guide the discussion.

Once there is a clear idea of the ways that RPM could be applied to improve application performance, efficiency or resiliency, it’ll be time to describe how the features of an RPM service are exposed to an application. That means taking a hard look at network APIs to be sure they export the functions and features that applications will need to access the service. The API is key, because it defines the ways that an application actually accesses a new network service. Keep in mind that such a service may or may not be a natural fit to existing applications; in some cases, it will fit naturally meaning that an existing application can easily begin to utilize the service to improve performance or efficiency. For other applications, more work will be needed to fully exploit the new service.

Notice that the development of the API is being driven from the top down by application requirements. This is a clear break from traditional network design, where the underlying network and its associated API are defined roughly in tandem. Contrast that to the approach being taken with RPM, where the set of desired network characteristics is described in terms of how an application will actually use the network. Interesting!

Armed with a clear sense of how an application might use Remote Persistent Memory and the APIs needed to access it, now’s the time for network architects and protocol designers to deliver enhanced network protocols and semantics that are best able to deliver the features defined by the new network APIs. And it’s time for hardware and software designers to get to work implementing the service and integrating it into server systems.

With all that in mind, here’s the current state of affairs for those who may be interested in participating. SNIA, through its NVM Programming Technical Working Group, has published a public document describing one very important use case for RPM – High Availability. The document describes the requirements that the SNIA NVM Programming Model – first released in December 2013 — might place on a high-speed network.  That document is available online. In keeping with the ‘top-down’ theme, SNIA’s work begins with an examination of the programming models that might leverage a Remote Persistent Memory service, and then explores the resulting impacts on network design. It is being used today to describe enhancements to existing APIs including both the Verbs API and the libfabric API.

In addition, SNIA and the OFA have established a collaboration to explore other use cases, with the idea that those use cases will drive additional API enhancements. That collaboration is just now getting underway and is taking place during open, bi-weekly meetings of the OFA’s OpenFabrics Interfaces Working Group (OFIWG). There is also a mailing list dedicated to the topic to which you can subscribe by going to www.lists.openfabrics.org and subscribing to the Ofa_remotepm mailing list.

And finally, we’ll be discussing the topic at the upcoming Flash Memory Summit, August 7-9, 2018.  Just go to the program section and click on the Persistent Memory major topic, and you’ll find a link to PMEM-202-1: Remote Persistent Memory.

See you in Santa Clara!

The OpenFabrics Alliance and the Pursuit of Efficient Access to Persistent Memory over Fabrics

 

Guest Columnist:  Paul Grun, Advanced Technology Development, Cray, Inc. and Vice-Chair, Open Fabrics Alliance (OFA)

Earlier this year, SNIA hosted its one-day Persistent Memory Summit in San Jose; it was my pleasure to be invited to participate by delivering a presentation on behalf of the OpenFabrics Alliance.  Check it out here.

The day long Summit program was chock full of deeply technical, detailed information about the state of the art in persistent memory technology coupled with previews of some possible future directions this exciting technology could conceivably take.  The Summit played to a completely packed house, including an auxiliary room equipped with a remote video feed.  Quite the event! Read More

Around the World, It’s a Persistent Memory Summer

This summer, join SNIA as they evangelize members’ industry activity to advance the convergence of storage and memory.

SNIA is participating in the first annual European In-Memory Computing Summit, June 20-21, 2017 at the Movenpick Hotel in Amsterdam.  SNIA Europe Vice-Chair and SNIA Solid State Storage Initiative (SSSI) Co-Chair Alex McDonald of NetApp keynotes a session on SNIA and Persistent Memory, highlighting Read More

Your Questions Answered on Non-Volatile DIMMs

 

by Arthur Sainio, SNIA NVDIMM SIG Co-Chair, SMART Modular

SNIA’s Non-Volatile DIMM (NVDIMM) Special Interest Group (SIG) had a tremendous response to their most recent webcast:  NVDIMM:  Applications are
Here
!  You can view the webcast on demand.

Viewers had many questions during the webcast.  In this blog, the NVDIMM SIG answers those questions and shares the SIG’s knowledge of NVDIMM technology. Read More

Not a Tempest But a Seachange – Persistent Memory

by Marty Foltyn

Persistent memory discussions are capturing the minds of SNIA members and colleagues.  At last month’s SNIA Storage Developer Conference, NVM (non-volatile memory) and NVMe sessions were standing-room-only, and opinion sharing continued into animated hallway discussions.  I encourage you to check out the many presentations on the SNIA SDC website, and to download the live recordings of the keynotes here.

memconSNIA continued their education on persistent memory at this week’s Memcon in Santa Clara CA. SNIA’s booth was packed with attendees asking questions like what is the difference between the different kinds of NVDIMMs (you’ll want to check out our new snia_nvdimm_infographic), and is NVDIMM a standard (indeed, it is, JEDEC just released the DDR4 NVDIMM-N Design Standard Revision 1.0 last month, and you can download the link from our website). Read More

SNIA NVM Summit Delivers the Persistent Memory Knowledge You Need

by Marty Foltyn

The discussion, use, and application of Non-volatile Memory (NVM) has come a long way from the first SNIA NVM Summit in 2013.  The significant improvements in persistent memory, with enormous capacity, memory-like speed and non-volatility, will make the long-awaited promise of the convergence storage and memory a reality. In this 4th annual NVM Summit, we will see how Storage and Memory have now converged, and learn that we are now faced with developing the needed ecosystem.  Register and join colleagues on Wednesday, January 20, 2016 in San Jose, CA to learn more, or follow http://www.snia.org/nvmsummit to review presentations post- event.

The Summit day begins with Rick Coulson, Senior Fellow, Intel, discussing the most recent developments in persistent memory with a presentation on All the Ways 3D XPoint Impacts Systems Architecture.

Ethan Miller, Professor of Computer Science at UC Santa Cruz, will discuss Rethinking Benchmarks for Non-Volatile Memory Storage Systems. He will describe the challenges for benchmarks posed by the transition to NVM, and propose potential solutions to these challenges.

Ken Gibson, NVM SW Architecture, Intel will present Memory is the New Storage: How Next Generation NVM DIMMs will Enable New Solutions That Use Memory as the High-Performance Storage Tier . This talk reviews some of the decades-old assumptions that change for suppliers of storage and data services as solutions move to memory as the new storage

Jim Handy, General Director, Objective Analysis, and Tom Coughlin, President, Coughlin Associates will discuss Future Memories and Today’s Opportunities, exploring the role of NVM in today’s and future applications. They will give some market analysis and projections for the various NVM technologies in use today.

Matt Bryson, SVP-Research, ABR, will lead a panel on NVM Futures-Emerging Embedded Memory Technologies, exploring the current status and future opportunities for NVM technologies and in particular both embedded and standalone MRAM technologies and associated applications.

Edward Sharp, Chief, Strategy and Technology, PMC-Sierra, will present Changes Coming to Architecture with NVM. Although the IT industry has made tremendous progress innovating up and down the computing stack to enable, and take advantage of, non-volatile memory, is it sufficient, and where are the weakest links to fully unlock the potential of NVM.

Don Jeanette, VP and John Chen, VP of Trendfocus will review the Solid State Storage Market, discuss what is happening in various segments, and why, as it relates to PCIe.

Dejan Vucinc, HGST San Jose Research Center will discuss Latency in Context: Finding Room for NVMs in the Existing Software Ecosystem. HGST Research has been working diligently to find out where is there room in the existing hardware/software ecosystem for emerging NVM technology when viewed as block storage rather than main memory. Vucinc will show an update on previously published results using prototype PCI Express-attached PCM SSDs and our custom device protocol, DC Express, as well as measurements of its latency and performance through a proper device driver using several different kinds of Linux kernel block layer architecture.

Arthur Sainio, Director Marketing, SMART Modular and Co-Chair, SNIA NVDIMM SIG, will lead a panel on NVDIMM. discussing how new media types are joining NAND Flash, and enhanced controllers and networking are being developed to unlock the latency and throughput advantages of NVDIMM.

Neal Christiansen, Principal Development Lead, Microsoft, Microsoft will discuss Storage Class Memory Support in the Windows OS. Storage Class Memories (SCM) have been the topic of R&D for the last few years and with the promise of near term product delivery, the question is how will Windows be enabled for such SCM products and how can applications take advantage of these capabilities.

Jeff Moyer, Principal Software Engineer, Red Hat will give an overview of the current state of Persistent Memory Support in the Linux Kernel.

Cristian Diaconu, Principal Software Engineer, Microsoft will present Microsoft SQL Hekaton – Towards Large Scale Use of PM for In-memory Databases, using the example of Hekaton (Sql Server in-memory database engine) to break down the opportunity areas for non-volatile memory in the database space.

Tom Talpey, Architect File Server Team, Microsoft, will discuss Microsoft Going Remote at Low Latency: A Future Networked NVM Ecosystem. As new ultra-low latency storage such as Persistent Memory and NVM is deployed, it becomes necessary to provide remote access – for replication, availability and resiliency to errors.

Kevin Deierling, VP Marketing, Mellanox will discuss the role of the network in developing Persistent Memory over Fabrics, and what are the key goals and key fabric features requirements.

Upcoming December 11 Webcast: Flash Memory Enables 4K and Beyond Video Workflows

by Marty Foltyn

The Consumer Electronics Show (CES), held each year in early January in Las Vegas, has moved a long way from the days when you had to search high and low on the show floor for storage-related exhibits. Step on the floor in 2016, and you may never get past the automobile displays which have the capability to track and store your every activity. And even if you do, the plethora of accessible tech, video imaging, and smart home apps will make your head spin!

Solid State Storage is an important contributor to the internet of things featured at CES, and understanding it is key to making informed choices. Get ready for CES 2016 by first attending a SNIA Solid State Storage webcast on Friday, December 11 at 11:00 am Pacific where Tom Coughlin, CEO of analyst firm Coughlin Associates, presents Flash Memory Enables 4K and Beyond Video Workflows.

As the price and availability of flash memory grows flash memory will enable future generations of media that is even more immersive than today as video moves to 8K and virtual reality begins to play an increasing role in entertainment. Tom will discuss how, as the resolution and frame rate for video increase, flash memory is staring to play a significant role for content capture, post production and content delivery. His presentation will include material from the 2015 Digital Storage in Media and Entertainment Report from Coughlin Associates (and the associated 2015 digital media professional survey) on the growing use of flash memory in all aspects of professional media and entertainment and put flash use in context with other storage technologies in this industry.

The webcast is an important lead in to the CES partner program Storage Visions Conference January 3-4 in Las Vegas, where SNIA will exhibit  solid state and persistent memory and have a pre-conference education day.  Register for this informative SNIA Brighttalk webcast , held on December 11, 2015 at 11:00 apm Pacific/2:00 pm Eastern at https://www.brighttalk.com/webcast/663/180197

SNIA’s Solid State Storage Initiative Advances the Industry at Flash Memory Summit

A classic case of SNIA Solid State Storage Initiative (SSSI) member collaboration for industry advancement was on display in the SSSI booth for NVDIMM-N demonstration at the Flash Memory Summit (FMS) 2015. Under the direction of SSSI Chair Jim Ryan and coordinated by NVDIMM SIG co chairs Arthur Sainio and Jeff Chang and TechDev Committee chair Eden Kim, the SSSI was able to update and include NVDIMM-N storage performance in the SSSI marketing collaterals on the Summary Performance Comparison by Storage Class charts.

2015SummaryPerformanceChart.NVDIMM.1200

Five SSSI member companies – AgigA Tech, Calypso, Micron, SMART Modular, and Viking Technology – collaborated over a four week period on the introduction of a new NVDIMM-N storage performance demonstration. While it is rare to have potential competitors collaborate in such a fashion, NVDIMM-N storage represents a new paradigm for super fast, low latency, high IO/watt storage solutions. The NVDIMM-SIG has taken a leadership position by evangelizing the technology and developing the industry infrastructure necessary for large scale deployment.

This collaboration highlighted a classic blend of technical, marketing and industry association cooperation.

In the weeks leading up to FMS, the NVDIMM-SIG planned for an in-booth demonstration of the NVDIMM-N storage modules. To pave the way for universal adoption, the team worked together to dial in the Intel Open Source block IO development driver to meet the standards of the SNIA Performance Test Specification (PTS). An added goal was inclusion of NVDIMM-N modules as a new line item on the Summary Performance Comparison by Storage Class chart which lists PTS performance for various storage technologies. Under the guidance of NVDIMM-SIG, a rush project was instigated to get NVDIMM-N performance data tested to the PTS for the trade show.

Micron took the lead by lending a Supermicro server with Micron NVDIMM-N to Calypso for testing. Calypso then installed CTS test software on the server to allow full testing to the PTS. Viking and SMART Modular contributed by helping dial in the drivers, as well as sending modules from Viking and SMART Modular to cross reference with the Micron modules. The test plan was comprised of several test iterations using single, dual and finally quad modules using each of the vendor contributed modules.

The early single and dual module tests ran into repeatability and stability issues. NVDIMM-SIG consulted with Intel on the nuance of the Intel block IO driver while Calypso continued testing. The team successfully completed a test run that met the PTS steady state requirements on the quad module in time to release data for the show.

We had a solid demonstration at the SNIA SSSI Flash Memory Summit Booth on NVDIMM-N Performance complete with marketing collateral available for review and a handout. NVDIMM-SIG members responded to the many questions and interest in the NVDIMM-N storage technology.

fms booth

“Once again,” said SSSI Chair Jim Ryan, “we can see the value and benefit of SNIA SSSI to its members, the SNIA educational community and the NVDIMM industry. I believe this is a great case study in how we all can contribute and benefit from working within the SSSI for the betterment of individual companies, market development and the Solid State Storage industry at large.” SSSI provides educational and marketing materials free of charge on its public website while SNIA SSSI members may join the NVDIMM-SIG and other SSSI committees. Anyone interested to find out more about the SSSI or any of its many committees can go to the following link http://www.snia.org/sssi.

 

Data Recovery and Selective Erasure of Solid State Storage a New Focus at SNIA

The rise of solid state storage has been incredibly beneficial to users in a variety of industries. Solid state technology presents a more reliable and efficient alternative to traditional storage devices. However, these benefits have not come without unforeseen drawbacks in other areas. For those in the data recovery and data erase industries, for example, solid state storage has presented challenges. The obstacles to data recovery and selective erasure capabilities are not only a problem for those in these industries, but they can also make end users more hesitant to adopt solid state storage technology.

Recently a new Data Recovery and Erase Special Interest Group (SIG) has been formed within the Solid State Storage Initiative (SSSI) within the Storage Networking Industry Association (SNIA). SNIA’s mission is to “lead the storage industry worldwide in developing and promoting standards, technologies and educational services to empower organizations in the management of information.” This fantastic organization has given the Data Recovery and Erase SIG a solid platform on which to build the initiative.

The new group has held a number of introductory open meetings for SNIA members and non-members to promote the group and develop the group’s charter. For its initial meetings, the group sought to recruit both SNIA members and non-members that were key stakeholders in fields related to the SIG. This includes data recovery providers, erase solution providers and solid state storage device manufacturers. Aside from these groups, members of leading standards bodies and major solid state storage device consumers were also included in the group’s initial formation.

The group’s main purpose is to be an open forum of discussion among all key stakeholders. In the past, there have been few opportunities for representatives from different industries to work together, and collaboration had often been on an individual basis rather than as a group. With the formation of this group, members intend to cooperate between industries on a collective basis in order to foster a more constructive dialogue incorporating the opinions and feedback of multiple parties.

During the initial meetings of the Data Recovery and Erase SIG, members agreed on a charter to outline the group’s purpose and goals. The main objective is to foster collaboration among all parties to ensure consumer demands for data recovery and erase services on solid state storage technology can be performed in a cost-effective, timely and fully successful manner

In order to achieve this goal, the group has laid out six steps needed, involving all relevant stakeholders:

  1. Build the business case to support the need for effective data recovery and erase capabilities on solid state technology by using use cases and real examples from end users with these needs.
  2. Create a feedback loop allowing data recovery providers to provide failure information to manufacturers in order to improve product design.
  3. Foster cooperation between solid state manufacturers and data recovery and erase providers to determine what information is necessary to improve capabilities.
  4. Protect sensitive intellectual property shared between data recovery and erase providers and solid state storage manufacturers.
  5. Work with standards bodies to ensure future revisions of their specifications account for capabilities necessary to enable data recovery and erase functionality on solid state storage.
  6. Collaborate with solid state storage manufacturers to incorporate capabilities needed to perform data recovery and erase in product design for future device models.

The success of this special interest group depends not only on the hard work of the current members, but also in a diverse membership base of representatives from different industries. We will be at Flash Memory Summit in booth 820 to meet you in person! Or you can visit our website at www.snia.org/forums/sssi for more information on this new initiative and all solid state storage happenings at SNIA.   If you’re a SNIA member and you’d like to learn more about the Data Recovery/Erase SIG or you think you’d be a good fit for membership, we’d love to speak with you.  Not a SNIA member yet? Email marty.foltyn@snia.org for details on joining.

New SNIA SSSI Webcast May 28 on Persistent Memory Advances

Join the NVDIMM Special Interest Group for an informative SNIA Brighttalk webcast on Persistent Memory Advances:  Solutions with Endurance, Performance & Non-Volatility on Thursday, May 28, 2015 at 12:00 noon Eastern/9:00 am Pacific.  Register at http://www.snia.org/news_events/multimedia#webcasts

Mario Martinez of Netlist, a SNIA SSSI NVDIMM SIG member, will discuss how persistent memory solutions deliver the endurance and performance of DRAM coupled with the non-volatility of Flash. This webinar will also update you on the latest solutions for enterprise server and storage designs, and provide insights into future persistent memory advances. A specific focus will be NVDIMM solutions, with examples from the member companies of the SNIA NVDIMM Special Interest Group.